DEV Community: Jose Roman Martin Gil The latest articles on DEV Community by Jose Roman Martin Gil (@rmarting). https://dev.to/rmarting https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F536848%2F106e6400-7095-4c47-9bc6-1b719de2a003.png DEV Community: Jose Roman Martin Gil https://dev.to/rmarting en ๐Ÿ“› Improving a GitHub Repo (II)! Jose Roman Martin Gil Tue, 20 Jun 2023 07:00:00 +0000 https://dev.to/rmarting/improving-a-github-repo-ii-17nb https://dev.to/rmarting/improving-a-github-repo-ii-17nb <p>My first post of <a href="https://blog.jromanmartin.io/2023/06/12/Improving-a-gh-repository.html">๐Ÿ“› Improving a GitHub Repo</a> describes many good things to add in any GitHub repository to be more productive and professional. However, that stuff can be hard to do every time a new repository is created, and we can forget to add something great. I found a way to accelerate this process and also do not forget to add anything great: <a href="https://docs.github.com/en/repositories/creating-and-managing-repositories/creating-a-template-repository">GitHub Repository templates</a>.</p> <p>GitHub template repository is the best way to replicate an standard structure, including folders, documentation, workflows, branches, and any file required to set up a new project. Using this pattern we can homogenize the structure of any repository of your organization, or also your own projects, easily and saving a lot of time. If you need to standardize your projects, or you need to create many projects on demand, definitely a template repository is your tool.</p> <p>As summary, the most great benefits of using repository template I found are:</p> <ul> <li>โŒ› Spend less time repeating code</li> <li>๐ŸŒŸ Focus on building new things</li> <li>๐Ÿฆพ Less manual configuration</li> <li>๐Ÿ“ Sharing boilerplate code across the code base</li> </ul> <p>And, the main features of a repository template are:</p> <ul> <li>Copy the entire repository files to a brand new repository</li> <li>Every template has a new url endpoint called <code>/generate</code> </li> <li>Share repository template through your organization or other GitHub users</li> </ul> <h2> My own repository template </h2> <p>I create mw own GitHub Template repository in here: <a href="https://github.com/rmarting/gh-repo-template">https://github.com/rmarting/gh-repo-template</a> including all the things described in my previous <a href="https://blog.jromanmartin.io/2023/06/12/Improving-a-gh-repository.html">post</a>, or new things added along the time.</p> <p>My template repository includes things such as:</p> <ul> <li>Initial content files aligned with the common patterns in any Open Source project: Contribution guide, Code of Conduct, contributors, โ€ฆ</li> <li>GitHub templates to report issues or open Pull Requests.</li> <li>Standard badges to summarize the repository.</li> <li>Standard workflows to release versions, or to implement Continuous Integration pipelines</li> </ul> <p>So, creating a new repository and setting up it takes few seconds and steps. Amazing!!!</p> <p>Do you have ideas, or comments about how to improve a template repository? Looking forward to hearing you with more contributions into my template repo, or adding comments in this post.</p> <p>๐Ÿค–๐Ÿšฉ Happy creation of new projects!!! ๐Ÿค–๐Ÿšฉ</p> <p><a href="http://blog.jromanmartin.io/images/bitmoji/happy-coding.avif"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--nGbPhloC--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/http://blog.jromanmartin.io/images/bitmoji/happy-coding.avif" alt="" title="Happy coding!!!" width="398" height="398"></a></p> community github git productivity ๐Ÿ–ญ How to resize a virtual disk Jose Roman Martin Gil Fri, 16 Jun 2023 07:00:00 +0000 https://dev.to/rmarting/how-to-resize-a-virtual-disk-3a6g https://dev.to/rmarting/how-to-resize-a-virtual-disk-3a6g <p>If you work with VMs it is very common that sometimes you need more space, but your VMs were defined with an estimated size. I started to use Virtual Machine Manager to manage my VMs when I joined to Red Hat (sorry but in my previous life I usually used Oracle VM VirtualBox) and sometimes I need to resize my image files but I didnโ€™t know how to do it.</p> <p>Thanks to <a href="https://github.com/oarribas">Oscar Arribas Arribas</a> I learned to do it using a few<code>virt-xxx</code> commands. It is very possible to do it using other commands/steps/alternatives however this way is good for me.</p> <h2> Step 0๏ธโƒฃ - Checking current VM disk size </h2> <p>Inside of your VM you can check the size of the each disk with the <code>df</code> command:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>[rhmw@f38mw01 ~]$ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 4.0M 0 4.0M 0% /dev tmpfs 977M 0 977M 0% /dev/shm tmpfs 391M 1.3M 390M 1% /run /dev/vda3 19G 4.7G 14G 26% / tmpfs 977M 40K 977M 1% /tmp /dev/vda3 19G 4.7G 14G 26% /home /dev/vda2 974M 257M 650M 29% /boot tmpfs 196M 56K 196M 1% /run/user/42 tmpfs 196M 40K 196M 1% /run/user/1000 </code></pre> </div> <p>Here the <code>home</code> has 20G allocated. I would like to extend it to 40G.</p> <h2> Step 1๏ธโƒฃ - Creating a new disk image </h2> <p>Your VM must be stopped before starting to resize it using a new disk image with the desired size.</p> <p>We can create a new disk using the <code>qemu-img</code> tool, something like this:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>on ๐ŸŽฉ โฏ qemu-img create -f qcow2 f38mw01-resized.qcow2 40G Formatting 'f38mw01-resized.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=42949672960 lazy_refcounts=off refcount_bits=16 </code></pre> </div> <p>Or creating the new image file by the Storage tab in the <code>virt-manager</code> Connection Details option (Edit -&gt; Connection Details):</p> <p><a href="http://blog.jromanmartin.io/images/2023/06/vm/vm-resize.avif"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--KZUy4LCH--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/http://blog.jromanmartin.io/images/2023/06/vm/vm-resize.avif" alt="" title="New disk image with more space" width="800" height="635"></a></p> <h2> Step 2๏ธโƒฃ - Renaming the old disk image </h2> <p>Rename the old image file as a backup file (it could be needed to use in a roll-back case):<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>mv f38mw01.qcow2 f38mw01.qcow2.backup </code></pre> </div> <p>You can also describe the file systems in the old image file:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>on ๐ŸŽฉ โฏ sudo virt-filesystems --long -h --all -a f38mw01.qcow2.backup Name Type VFS Label MBR Size Parent /dev/sda1 filesystem unknown - - 1.0M - /dev/sda2 filesystem ext4 - - 973M - /dev/sda3 filesystem btrfs fedora_localhost-live - 19G - btrfsvol:/dev/sda3/home filesystem btrfs fedora_localhost-live - - - btrfsvol:/dev/sda3/root filesystem btrfs fedora_localhost-live - - - btrfsvol:/dev/sda3/root/var/lib/machines filesystem btrfs fedora_localhost-live - - - /dev/sda1 partition - - - 1.0M /dev/sda /dev/sda2 partition - - - 1.0G /dev/sda /dev/sda3 partition - - - 19G /dev/sda /dev/sda device - - - 20G - </code></pre> </div> <h2> Step 3๏ธโƒฃ - Truncating the new disk image </h2> <p>Truncate the old image file and resize the new image file with the new space:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>on ๐ŸŽฉ โฏ sudo truncate -r f38mw01.qcow2.backup f38mw01-resized.qcow2 on ๐ŸŽฉ โฏ sudo truncate -s +20G f38mw01-resized.qcow2 </code></pre> </div> <h2> Step 4๏ธโƒฃ - Expanding the new disk image </h2> <p>Expand the new image file using as base the old image file. In this step I am expanding the physical disk mounted for the <code>home</code> folder.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>on ๐ŸŽฉ โฏ sudo virt-resize --expand /dev/sda3 f38mw01.qcow2.backup f38mw01-resized.qcow2 [0.0] Examining f38mw01.qcow2.backup ********** Summary of changes: virt-resize: /dev/sda1: This partition will be left alone. virt-resize: /dev/sda2: This partition will be left alone. virt-resize: /dev/sda3: This partition will be resized from 19.0G to 39.0G. The filesystem btrfs on /dev/sda3 will be expanded using the โ€˜btrfs-filesystem-resizeโ€™ method. ********** [2.6] Setting up initial partition table on f38mw01-resized.qcow2 [13.4] Copying /dev/sda1 [13.4] Copying /dev/sda2 100% โŸฆโ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โŸง --:-- [15.8] Copying /dev/sda3 100% โŸฆโ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โŸง 00:00 [48.0] Expanding /dev/sda3 using the โ€˜btrfs-filesystem-resizeโ€™ method virt-resize: Resize operation completed with no errors. Before deleting the old disk, carefully check that the resized disk boots and works correctly. </code></pre> </div> <h2> Step 5๏ธโƒฃ - Starting the VM with the new disk image </h2> <p>Rename the new disk image as the original one used by the VM:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>on ๐ŸŽฉ โฏ mv f38mw01-resized.qcow2 f38mw01.qcow2 </code></pre> </div> <p>Start the VM and the check that our <code>home</code> has more space:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>[rhmw@f38mw01 ~]$ df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 4.0M 0 4.0M 0% /dev tmpfs 977M 0 977M 0% /dev/shm tmpfs 391M 1.3M 390M 1% /run /dev/vda3 39G 4.7G 34G 13% / tmpfs 977M 40K 977M 1% /tmp /dev/vda2 974M 257M 650M 29% /boot /dev/vda3 39G 4.7G 34G 13% /home tmpfs 196M 56K 196M 1% /run/user/42 tmpfs 196M 40K 196M 1% /run/user/1000 </code></pre> </div> <p>The <code>/dev/vda3</code> now is 34G (in the step 0, the size was 19G). Great!!!</p> <h2> Bonus Track ๐Ÿ’ก - Resizing Microsoft Windows VMs </h2> <p>I know, I know what you are thinking ๐Ÿค” โ€ฆ this stuff works because I am using a Linux OS ๐Ÿ˜‡. However, this process also works for Windows VMs.</p> <p>Here an example of a Windows 10 with a hard disk of 40G to extend to 50G:</p> <p><a href="http://blog.jromanmartin.io/images/2023/06/vm/vm-win10-40g.avif"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--KyLHjNte--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/http://blog.jromanmartin.io/images/2023/06/vm/vm-win10-40g.avif" alt="" title="40G in my hard disk!" width="359" height="493"></a></p> <p>The process is exactly the same:</p> <p>Create new disk image:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>on ๐ŸŽฉ โฏ qemu-img create -f qcow2 win10-resized.qcow2 50G Formatting 'win10-resized.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=53687091200 lazy_refcounts=off refcount_bits=16 </code></pre> </div> <p>Back the original disk image:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>on ๐ŸŽฉ โฏ mv win10.qcow2 win10.qcow2.backup </code></pre> </div> <p>Check the file systems of the old image:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>on ๐ŸŽฉ โฏ sudo virt-filesystems --long -h --all -a win10.qcow2.backup Name Type VFS Label MBR Size Parent /dev/sda1 filesystem ntfs System Reserved - 579M - /dev/sda2 filesystem ntfs - - 39G - /dev/sda1 partition - - 07 579M /dev/sda /dev/sda2 partition - - 07 39G /dev/sda /dev/sda device - - - 40G - </code></pre> </div> <p>Truncate the new disk image:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>on ๐ŸŽฉ โฏ sudo truncate -r win10.qcow2.backup win10-resized.qcow2 on ๐ŸŽฉ โฏ sudo truncate -s +10G win10-resized.qcow2 </code></pre> </div> <p>Expand the new disk image:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>on ๐ŸŽฉ โฏ sudo virt-resize --expand /dev/sda2 win10.qcow2.backup win10-resized.qcow2 [0.0] Examining win10.qcow2.backup ********** Summary of changes: virt-resize: /dev/sda1: This partition will be left alone. virt-resize: /dev/sda2: This partition will be resized from 39.4G to 49.4G. The filesystem ntfs on /dev/sda2 will be expanded using the โ€˜ntfsresizeโ€™ method. ********** [1.9] Setting up initial partition table on win10-resized.qcow2 [2.8] Copying /dev/sda1 [3.6] Copying /dev/sda2 100% โŸฆโ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โ–’โŸง 00:00 [55.7] Expanding /dev/sda2 using the โ€˜ntfsresizeโ€™ method virt-resize: Resize operation completed with no errors. Before deleting the old disk, carefully check that the resized disk boots and works correctly. </code></pre> </div> <p>Rename the new disk using the original name<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>on ๐ŸŽฉ โฏ mv win10-resized.qcow2 win10.qcow2 </code></pre> </div> <p>Start the VM and check the new disk size:</p> <p><a href="http://blog.jromanmartin.io/images/2023/06/vm/vm-win10-50g.avif"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--f7AjOzVU--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/http://blog.jromanmartin.io/images/2023/06/vm/vm-win10-50g.avif" alt="" title="Now 50G in my hard disk!" width="366" height="499"></a></p> <p>๐Ÿšฉ Happy resizing!!! ๐Ÿค–</p> <p><a href="http://blog.jromanmartin.io/images/bitmoji/the-end.avif"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--MuoFnar_--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/http://blog.jromanmartin.io/images/bitmoji/the-end.avif" alt="" title="That's all!!!" width="398" height="398"></a></p> community productivity tools howto ๐Ÿ“› Improving a GitHub Repo! Jose Roman Martin Gil Mon, 12 Jun 2023 07:00:00 +0000 https://dev.to/rmarting/improving-a-github-repo-2o55 https://dev.to/rmarting/improving-a-github-repo-2o55 <p>I have been using <a href="https://github.com">GitHub</a> for a long time and I spent time on a daily basis reviewing repos in the Open Source space. One of the most important things , from my point of view, is to get a good overview of the repository, a good documentation, but also good highlights, such as releases, status of the project, Changelogs, Contribution Guides, emojis (<a href="https://blog.jromanmartin.io/2020/09/28/why-i-use-emoji-in-my-git-commits.html">why not?</a>) โ€ฆ so I can get faster a good summary of the repository. This is not easy and there are many different ways to do it, but I found some of them very easy to add in any repository.</p> <p>This post covers two of these mechanisms to improve any GitHub Repository:</p> <ul> <li>๐Ÿ“› Repository Badges</li> <li>โœ… Changelogs and ๐Ÿค–๐Ÿšฉautomatic releasing process</li> </ul> <h2> ๐Ÿ“› Repository Badges </h2> <p>How does it look like a repo with badges? Something like this:</p> <p><a href="http://blog.jromanmartin.io/images/2023/06/github/gh-repo-badges.avif"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--HJcw_vfJ--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/http://blog.jromanmartin.io/images/2023/06/github/gh-repo-badges.avif" alt="" title="GitHub repo with badges" width="461" height="161"></a></p> <p>Nice ๐Ÿซถ, right?</p> <p>Badges are an easy way to summarize a repo with information about topics such as building, test results, license, pipelines or workflows, versions, โ€ฆ This information provides quality metadata coming from many different resources. So meanwhile you are browsing, you get all this information in a single view. Incredible!</p> <p>I found a simple way to integrate almost any badge in my repository โ€ฆ<a href="https://shields.io/">Shields.io</a>. It is a service providing badges in different formats to integrate in GitHub readme files. This service supports a bunch of continuous integration services, package registries, distributions, app stores, social networks, code coverage services, and code analysis services (anything else? ๐Ÿคท๐Ÿฝโ€โ™€๏ธ).</p> <p>In short, using the web site you can customize your badge to your own requisites and 3rd party services, getting a code to add in your GitHub readme easily.</p> <p>For example, the previous image is rendered using the next entries in my readme file of my Blog site repository:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>![License](https://img.shields.io/github/license/rmarting/rmarting.github.io?style=plastic) ![Main Lang](https://img.shields.io/github/languages/top/rmarting/rmarting.github.io) ![Languages](https://img.shields.io/github/languages/count/rmarting/rmarting.github.io) ![Last Commit](https://img.shields.io/github/last-commit/rmarting/rmarting.github.io) </code></pre> </div> <p>Easy โœ…, and powerful ๐Ÿ’ช! So, donโ€™t forget to add your badges in your repo to help me, and others ๐Ÿค—!</p> <h2> โœ… Changelogs and ๐Ÿค–๐Ÿšฉautomatic releasing process </h2> <p>Changelog, as a comprehensive and up-to-date file, is crucial for effective project management and collaboration. A changelog serves as a documented record of all the notable changes, enhancements, and bug fixes made to your software over time. It not only provides transparency and accountability but also facilitates communication among team members and external contributors. This file enables users and developers to easily track the evolution of the project, understand the latest features and improvements, and quickly identify any potential issues or compatibility concerns.</p> <p>Getting all these benefits require a regular updating of that file, usually after releasing a new version or iteration of our software. But, how to track all the changes between versions? Who should do it? When? โ€ฆ It seems that it could be tedious every time if we have to do it manually โ€ฆ we can forget something to add, or we can forget to update it at all.</p> <p>As fan of โ€ฆ</p> <p><a href="https://www.redbubble.com/i/sticker/AUTOMATE-ALL-THE-THINGS-by-antonwadstrom/29760692.EJUG5"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--ckbtS9A2--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/http://blog.jromanmartin.io/images/2023/06/github/automate-all-the-things.avif" alt="" title="Automate all the things (Sticker)" width="795" height="443"></a></p> <p>There is a way to automatically update the changelog file every time a new version is released. This blog post summarizes this process.</p> <p><strong>Step 1๏ธโƒฃ - Create your Changelog file</strong></p> <p>Create a file, usually called <code>CHANGELOG.md</code>, with the following content:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code># Changelog All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] </code></pre> </div> <p>To delve deeper into the significance of changelog files and learn about best practices for creating and maintaining them, I recommend checking out <a href="https://keepachangelog.com/">Keep a Changelog</a>. This resource offers a comprehensive guide and industry-accepted standards for crafting informative and well-structured changelogs.</p> <p><strong>Step 2๏ธโƒฃ - Use a Release workflow to publish new releases</strong></p> <p>The <a href="https://github.com/marketplace/actions/release-drafter">Release Drafter GitHub Action</a> is an incredible GitHub action to automate a new release of the repository. The action is initially designed to draft a new release, but it is also valid to release automatically the version. In my case, I will automatically release the version as soon as a new tag is pushed.</p> <p>The following <code>release-drafter.yml</code> file inside of the <code>.github/workflows</code> folder will publish a new release after a new tag is pushed into the repository. The tag must be aligned with the<a href="https://semver.org">Semantic Versioning</a>:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>name: ๐Ÿ”– Release Drafter ๐Ÿ”– on: push: tags: - v[0-9]+.[0-9]+.[0-9]+ permissions: contents: read jobs: update_release_draft: name: Release drafter runs-on: ubuntu-latest permissions: # write permission is required to create a github release contents: write steps: - name: Update Release Draft uses: release-drafter/release-drafter@v5 with: publish: true prerelease: false env: # Instead of GITHUB_TOKEN Ref: https://github.com/stefanzweifel/changelog-updater-action/discussions/30 GITHUB_TOKEN: $ </code></pre> </div> <p>The <code>publish: true</code> attribute publishes the release as final, just because the <code>prerelease</code> attribute is marked as <code>false</code>.</p> <p><strong>Step 3๏ธโƒฃ - Format the Release content</strong></p> <p>The content of the release will include information coming from the different pull request, issues, and commits. This information can be included automatically into the release notes using different patterns. These patterns are described in the <code>release-drafter.yml</code> file inside <code>.github</code> folder:</p> <p>The following example is a full example using different categories of information to add into the release notes.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code># This release drafter follows the conventions from https://keepachangelog.com name-template: 'v$RESOLVED_VERSION' tag-template: 'v$RESOLVED_VERSION' template: | ## What Changed ๐Ÿ‘€ $CHANGES **Full Changelog** : https://github.com/$OWNER/$REPOSITORY/compare/$PREVIOUS_TAG...v$RESOLVED_VERSION categories: - title: ๐Ÿš€ Features labels: - feature - enhancement - title: ๐Ÿ› Bug Fixes labels: - fix - bug - title: โš ๏ธ Changes labels: - changed - title: โ›”๏ธ Deprecated labels: - deprecated - title: ๐Ÿ—‘ Removed labels: - removed - title: ๐Ÿ” Security labels: - security - title: ๐Ÿ“„ Documentation labels: - docs - documentation - title: ๐Ÿงฉ Dependency Updates labels: - deps - dependencies collapse-after: 5 change-template: '* $TITLE (#$NUMBER)' change-title-escapes: '\&lt;*_&amp;' # You can add # and @ to disable mentions, and add ` to disable code blocks. exclude-labels: - skip-changelog </code></pre> </div> <p><strong>Step 4๏ธโƒฃ - Update Changelog file</strong></p> <p>After a new version is released, we want to update the changelog with the latest changes, as we are doing with the release notes. We can automate it using another amazing GitHub Action - <a href="https://github.com/marketplace/actions/changelog-updater">Changelog Updater</a>.</p> <p>This action can be integrated into another workflow (i.e: <code>update-changelog.yml</code> inside of the <code>.github/workflows</code> folder). This workflow can be similar to:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>name: ๐Ÿ“„ Update Changelog ๐Ÿ“„ on: release: types: [released] jobs: update: name: Update Changelog runs-on: ubuntu-latest permissions: # Give the default GITHUB_TOKEN write permission to commit and push the # updated CHANGELOG back to the repository. # https://github.blog/changelog/2023-02-02-github-actions-updating-the-default-github_token-permissions-to-read-only/ contents: write steps: - name: Checkout code uses: actions/checkout@v3 - name: Update Changelog uses: stefanzweifel/changelog-updater-action@v1 with: latest-version: $ release-notes: $ - name: Commit updated Changelog uses: stefanzweifel/git-auto-commit-action@v4 with: branch: main commit_message: '๐Ÿ”– Update changelog' file_pattern: CHANGELOG.md </code></pre> </div> <p>So, this workflow will start when a new release is released (<code>types: [released]</code>), including the changes from previous release and committing the change into the <code>main</code> branch of our repo.</p> <p><strong>Step 5๏ธโƒฃ - Linking release and update changelog workflows</strong></p> <p>There is an issue reported <a href="https://github.com/stefanzweifel/changelog-updater-action/discussions/30">here</a>about how to automatically trigger the update changelog workflow from the release workflow. The workaround to fix it requires adding a new secret (i.e: <code>PERSONAL_ACCESS_TOKEN</code>) into your repo:</p> <p><a href="http://blog.jromanmartin.io/images/2023/06/github/gh-secrets.avif"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--8YgAVbKW--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/http://blog.jromanmartin.io/images/2023/06/github/gh-secrets.avif" alt="" title="GitHub Repo secrets" width="797" height="653"></a></p> <p><strong>Step 6๏ธโƒฃ - Release a new version</strong></p> <p>Now, it is very simple, just follow your development workflow, using your pull-request life cycle, add the labels of your own repository, and then tag a new version when you are ready to do it.</p> <p>Push it into your repo:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>git tag v1.2.1 -m "Version 1.2.1" git push origin v1.2.1 </code></pre> </div> <p>The workflows run as expected:</p> <p><a href="http://blog.jromanmartin.io/images/2023/06/github/gh-actions.avif"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--md-3NGo8--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/http://blog.jromanmartin.io/images/2023/06/github/gh-actions.avif" alt="" title="Workflows executed" width="800" height="244"></a></p> <p>โ€ฆ a new release is created, including the notes:</p> <p><a href="http://blog.jromanmartin.io/images/2023/06/github/gh-new-release.avif"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--G8pTwVn6--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/http://blog.jromanmartin.io/images/2023/06/github/gh-new-release.avif" alt="" title="New GitHub Release" width="800" height="625"></a></p> <p>โ€ฆ and the changelog is updated too:</p> <p><a href="http://blog.jromanmartin.io/images/2023/06/github/gh-changelog.avif"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--UydO1M8P--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/http://blog.jromanmartin.io/images/2023/06/github/gh-changelog.avif" alt="" title="Changelog updated" width="800" height="450"></a></p> <p>This is โ€ฆ</p> <p><a href="http://blog.jromanmartin.io/images/bitmoji/super-awesome.avif"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--S5I3Yde9--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_800/http://blog.jromanmartin.io/images/bitmoji/super-awesome.avif" alt="" title="Super Awesome" width="398" height="398"></a></p> <p>๐Ÿค–๐Ÿšฉ Happy automating releasing!!! ๐Ÿค–๐Ÿšฉ</p> <h2> References </h2> <p>This blog post is my own summary about this process, but it was based from the content and experience of others, such as:</p> <ul> <li><a href="https://tiagomichaelsousa.dev/articles/stop-writing-your-changelogs-manually">Stop writing your changelogs manually</a></li> <li><a href="https://github.com/marketplace/actions/release-drafter">Release Drafter GitHub Action</a></li> <li><a href="https://github.com/marketplace/actions/changelog-updater">Changelog Update GitHub Action</a></li> </ul> <p>My kudos โค๏ธ to all of them!!!</p> community github git productivity Cloud Native CICD Pipelines in OpenShift Jose Roman Martin Gil Fri, 01 Apr 2022 00:00:00 +0000 https://dev.to/rmarting/cloud-native-cicd-pipelines-in-openshift-4eeg https://dev.to/rmarting/cloud-native-cicd-pipelines-in-openshift-4eeg <h2> Cloud Native CICD Pipelines in OpenShift </h2> <p>My first <a href="https://openpracticelibrary.com/practice/continuous-integration/">Continuous Integration</a> and<a href="https://openpracticelibrary.com/practice/continuous-delivery/">Continuous Delivery</a> pipelines (from now CICD) were created with <a href="https://en.wikipedia.org/wiki/Hudson_(software)">Hudson</a>(I know, I know !! I am very old ๐Ÿ‘ด on this space), and after that with <a href="https://www.jenkins.io/">Jenkins</a>for longer time. During this long period I used it (and others similar) to build, test, package, and deploy many different kind of applications (Monolith, SOA Services, Microservices, standalone apps, โ€ฆ) into many different kind of platforms (<a href="https://tomcat.apache.org/">Tomcat</a>, <a href="https://www.redhat.com/en/technologies/jboss-middleware/application-platform">Red Hat JBoss Enterprise Applications</a>,<a href="https://www.oracle.com/es/java/weblogic/">WebLogic</a>, โ€ฆ) and of course on containers platform such as <a href="https://www.redhat.com/en/technologies/cloud-computing/openshift">Red Hat OpenShift</a>.</p> <p>However, in cloud environments with cloud native applications in some cases I found so much complexity not easy to deal with it. Basically these tools were defined to run on Virtual Machines, required IT operations for maintenance, conflicts between teams or projects with shared plugins or extensions, no native interoperability with Kubernetes resources, โ€ฆ</p> <p>โ€ฆ and nowadays I found a new player in this scenario to improve my CICD pipelines in the new Cloud Native World, with containers, Kubernetes, and OpenShift. This player is <a href="https://tekton.dev/">Tekton</a>, or<a href="https://cloud.redhat.com/learn/topics/ci-cd">Red Hat OpenShift Pipelines</a> as the enterprise version for OpenShift.</p> <p>Tekton is a cloud-native solution for building CICD systems, providing a set of building blocks, components and an extended catalog (<a href="https://hub.tekton.dev/">Tekton Hub</a>) with great resources to use, making it a complete ecosystem. It is part of the <a href="https://cd.foundation/">CD Foundation</a> with a great community, very active.</p> <p>As Tekton is installed as a <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/operator/">Kubernetes Operator</a>, providing <a href="https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/">Custom Resources Definitions</a>to define the building blocks, it is very easy to create, and reuse them in the pipelines. As other Kubernetes or OpenShift objects, Tekton CRDs are first-citizens, so many of the processes uses to manage your OpenShift platform are valid for them. For example, as a fan of <a href="https://openpracticelibrary.com/practice/everything-as-code/">Everything as Code</a>practice, I can define my CICD pipelines as code and store them in a Git repository.</p> <p>Tekton uses the services provides by the OpenShift, so it is designed for containers, and scalability. It means that the pipelines and tasks are executed on-demand with containers, so it is easy to scale them. We, as CICD designers, donโ€™t need to deal with the platform, or infrastructure, as OpenShift provides us the services, and Tekton the objects to design the flow of our CICD pipeline.</p> <p>In that integration with OpenShift services, the building images processes are now really native and we could use any of the technologies available, such as <a href="https://github.com/openshift/source-to-image">source-to-image</a>,<a href="https://buildah.io/">buildah</a>, <a href="https://github.com/GoogleContainerTools/kaniko">kaniko</a>,<a href="https://github.com/GoogleContainerTools/jib">jib</a>, โ€ฆ Not more needed creating a custom image for a Jenkins-agent to build our application.</p> <p>The same to integrate the deployment processes of your application, as you can interact natively with the platform โ€ฆ but in this scenario I am more fan to move the Continuous Delivery following the <a href="https://openpracticelibrary.com/practice/gitops/">GitOps</a> approach with another amazing tool as <a href="https://github.com/argoproj/argo-cd">ArgoCD</a> (but that is another story, and other blog-post ๐Ÿ˜‰).</p> <p>At last, but not least, Tekton provides a set of amazing tooling to use on your favorite IDE, command-line tool and so on, and accelerate the adoption by your side, and make your life easier:</p> <ul> <li><a href="https://tekton.dev/docs/cli/"><code>tkn</code> command line interface</a></li> <li><a href="https://github.com/redhat-developer/vscode-tekton">Tekton Pipelines Extension for VSCode</a></li> <li><a href="https://plugins.jetbrains.com/plugin/14096-tekton-pipelines-by-red-hat">Tekton Pipelines by Red Hat for IntelliJ</a></li> </ul> <p>So, letโ€™s go through across the main components of this amazing project.</p> <h2> Tekton Components </h2> <p>Tekton provides a set of different components to design and build your pipelines:</p> <ul> <li>Tasks</li> <li>Pipelines</li> <li>Workspaces</li> <li>Triggers</li> </ul> <p>There are others too, but these ones are the base.</p> <h3> Tasks </h3> <p><a href="https://tekton.dev/docs/pipelines/tasks/"><code>Tasks</code></a> is a collection of <code>Steps</code> that you define and arrange in a specific order of execution as part of your continuous integration flow.</p> <p><code>Tasks</code> can have more than one <code>step</code>, allowing to specialize the task with more detailed steps. The steps will run in the order in which they are defined in the steps array.</p> <p>A <code>Task</code> is available within a specific namespace, while a <code>ClusterTask</code> is available across the entire cluster.</p> <p>A <code>Task</code> is executed as a Pod on your OpenShift cluster.</p> <p>This is the typical <code>Hello World</code> Task.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>apiVersion: tekton.dev/v1beta1 kind: Task metadata: name: hello-task spec: steps: - name: say-hello image: registry.redhat.io/ubi7/ubi-minimal command: ['/bin/bash'] args: ['-c', 'echo Hello World'] </code></pre> </div> <p>Meanwhile a <code>Task</code> is a definition, the execution of the task with the results and outputs is a <code>TaskRun</code>.</p> <p>An execution of previous task should be similar to (<em>simplified and omitted some fields</em>):<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>apiVersion: tekton.dev/v1beta1 kind: TaskRun metadata: generateName: hello-task-run- name: hello-task-run-9d8hs uid: f3c8d81b-3e8d-4ad5-a01b-bf9b147485f6 creationTimestamp: '2022-03-31T16:34:41Z' namespace: pipelines-demo labels: app.kubernetes.io/managed-by: tekton-pipelines tekton.dev/task: hello-task spec: taskRef: kind: Task name: hello-task status: completionTime: '2022-03-31T16:34:47Z' conditions: - lastTransitionTime: '2022-03-31T16:34:47Z' message: All Steps have completed executing reason: Succeeded status: 'True' type: Succeeded podName: hello-task-run-9d8hs-pod-kqcjv startTime: '2022-03-31T16:34:41Z' steps: - container: step-say-hello imageID: &gt;- registry.redhat.io/ubi7/ubi-minimal@sha256:700ec6f27ae8380ca1a3fcab19b5630d5af397c980628fa1a207bf9704d88eb0 name: say-hello terminated: containerID: cri-o://346b671912a63a98b310f0f06f0bcd9d9e3fab3b24a75246aed4921863b1d146 exitCode: 0 finishedAt: '2022-03-31T16:34:46Z' reason: Completed startedAt: '2022-03-31T16:34:46Z' </code></pre> </div> <p>OpenShift provides a great dashboard to browse and inspect the Tasks and TasksRun</p> <p><a href="http://blog.jromanmartin.io/images/ocp-pipelines/ocp-tasks-dashboard.png"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--C64qtOda--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://blog.jromanmartin.io/images/ocp-pipelines/ocp-tasks-dashboard.png" alt="" title="OpenShift Tasks Dashboard" width="880" height="373"></a></p> <h3> Pipelines </h3> <p><a href="https://tekton.dev/docs/pipelines/pipelines/"><code>Pipelines</code></a> are a collection of <code>Tasks</code> that you define and arrange in a specific order of execution as part of your continuous integration flow. In fact, tasks should do one single thing so you can reuse them across pipelines or even within a single pipeline.</p> <p>You can configure various execution conditions to fit your business needs.</p> <p>This <a href="https://github.com/rmarting/ocp-pipelines-demo/blob/main/05-say-things-in-order-pipeline.yaml">example</a> could give you a general view of a pipeline. This pipeline should be represented as:</p> <p><a href="http://blog.jromanmartin.io/images/ocp-pipelines/ocp-pipeline-flow.png"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--kkJaBZKs--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://blog.jromanmartin.io/images/ocp-pipelines/ocp-pipeline-flow.png" alt="" title="Pipeline Flow" width="428" height="102"></a></p> <p>Each <code>Task</code> in a <code>Pipeline</code> executes as a <code>Pod</code> on your OpenShift cluster.</p> <p>Meanwhile a <code>Pipeline</code> is a definition, the execution of the pipeline with the results and outputs is a <code>PipelineRun</code>.</p> <p>OpenShift provides a great dashboard to browse and inspect the Tasks and TasksRun</p> <p><a href="http://blog.jromanmartin.io/images/ocp-pipelines/ocp-pipelines-dashboard.png"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--_U2_Egzb--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://blog.jromanmartin.io/images/ocp-pipelines/ocp-pipelines-dashboard.png" alt="" title="OpenShift Pipelines Dashboard" width="880" height="355"></a></p> <h3> Workspaces </h3> <p><a href="https://tekton.dev/docs/pipelines/workspaces/"><code>Workspaces</code></a> allow <code>Tasks</code> to declare parts of the filesystem that need to be provided at runtime by <code>TaskRuns</code>. The main use cases are:</p> <ul> <li>Storage of inputs and/or outputs</li> <li>Sharing data among <code>Tasks</code> </li> <li>Mount points for configurations held in <code>Secrets</code> or <code>ConfigMaps</code> </li> <li>A cache of build artifacts that speed up jobs</li> </ul> <p><code>Workspaces</code> are similar to <code>Volumes</code> except that they allow a <code>Task</code> author to defer to users and their <code>TaskRuns</code> when deciding which class of storage to use.</p> <h3> Triggers </h3> <p><a href="https://tekton.dev/docs/triggers/"><code>Triggers</code></a> are the components ready to detect and extract information from events from a variety of sources and execute <code>Tasks</code> or <code>Pipelines</code> to respond them.</p> <p>Triggers are a set of different objects:</p> <ul> <li> <code>EventListener</code>: listens for events at a specified port on your OpenShift cluster. Specifies one or more <code>Triggers</code> or <code>TriggerTemplates</code>.</li> <li> <code>Trigger</code>: specifies what happens when the <code>EventListener</code> detects an event. It is defined with a <code>TriggerTemplate</code>, a <code>TriggerBinding</code>, and optionally, an <a href="https://tekton.dev/docs/triggers/interceptors/">Interceptor</a>.</li> <li> <code>TriggerTemplate</code>: specifies a blueprint for the resource, such as a <code>TaskRun</code> or <code>PipelineRun</code>, that you want to instantiate and/or execute when your <code>EventListener</code> detects an event.</li> <li> <code>TriggerBinding</code>: specifies the fields in the event payload from which you want to extract data and the fields in your corresponding <code>TriggerTemplate</code> to populate with the extracted values. You can then use the populated fields in the <code>TriggerTemplate</code> to populate fields in the associated <code>TaskRun</code> or <code>PipelineRun</code>.</li> </ul> <p>The most common use case of the <code>Triggers</code> and <code>EventListeners</code> is integrated with Git repositories through the use of WebHooks. A Git mechanism to get data from any change in a Git repository.</p> <h2> Show me the code </h2> <p>But, you could do more things with OpenShift Pipelines, to design you cloud native pipelines. This is a small briefing of the main characteristics and features. But if you want to play with this new toy, I created a sample <a href="https://github.com/rmarting/ocp-pipelines-demo">GitHub repository</a> with a demo of tasks, pipelines and triggers.</p> <p><a href="https://github.com/rmarting/ocp-pipelines-demo">https://github.com/rmarting/ocp-pipelines-demo</a></p> <p>From here, only your imagination, use cases and Tekton could help you to create amazing pipelines in a easy, descriptive and simple way.</p> <p>And if you want to dive deeper, don't miss to check the following references:</p> <ul> <li><a href="https://tekton.dev/docs/">Tekton Documentation</a></li> <li> <a href="https://pipelinesascode.com/">Pipelines as Code</a>, an opinionated CI based on OpenShift Pipelines / Tekton.</li> <li> <a href="https://github.com/tektoncd/chains">Tekton Chains</a> for supply chain security.</li> </ul> <p>Happy cloud-native pipelining ๐Ÿ˜ƒ!!!</p> tutorial openshift operators tekton Monorepo, GitOps, CICD and beyond Jose Roman Martin Gil Fri, 25 Mar 2022 10:00:00 +0000 https://dev.to/rmarting/monorepo-gitops-cicd-and-beyond-26a8 https://dev.to/rmarting/monorepo-gitops-cicd-and-beyond-26a8 <h1> GitOps Product Monorepo Sample </h1> <p>Developing cloud native products following Agile, and DevOps practices could require to use different approaches, patterns and processes to do it in a fast pace. Many of the most common patterns in this space are:</p> <ul> <li>Product Monorepo</li> <li>Trunk-based development</li> <li>GitOps</li> <li>Cloud Native Application</li> <li>Continuous Integration, Continuous Delivery and Continuous Deployment</li> <li>Sealed Secrets</li> </ul> <p>Use all of them at the same time could be a challenge for a new team, but it is possible getting the best benefits from each of them. I have working in many different use cases and products for a long time, many times using these patterns in somehow, learning from the pitfalls, and getting the best benefits. This repo represents an <em>opinionated</em> manner to do it for a new product team, combining all these practices in the same place.</p> <p>Hoping this approach could help in your use case. Use carefully all the assess, and land into your specif use case.</p> <p>The <code>product</code> is the software solution created for a business scenario, adding the value and solution to achieve the goals of the business. We build software to resolve business scenarios.</p> <p>โ— Soon a blog post will try to clarify many of the aspects related with this repo. Stay tunned!โ—</p> <h2> Product Monorepo </h2> <p>Product <a href="https://en.wikipedia.org/wiki/Monorepo">Monorepo</a> means to have everything related with a software product in one single place, shared with the product team, to implement the full Software Delivery Life cycle of the product.</p> <p>One of the most important topics for a Product Monorepo is to have a clear folder structure, otherwise you could get a chaotic layout with more pains that gains.</p> <p>This repo is organized as:</p> <ul> <li> <a href="//./bootstrap/README.md">bootstrap</a> folder includes the initial components to setup the environment, mainly tools to define the base of the rest of the solution.</li> <li> <a href="//./apps/">apps</a> folder includes the list of applications or components of the product.</li> <li> <a href="//./charts/README.md">charts</a> folder includes the Helm Charts to accelerate the deployment of any component, tool or application related with the product.</li> <li> <a href="//./argocd/README.md">argocd</a> folder includes the items related with ArgoCD and GitOps.</li> <li> <a href="//./tekton/README.md">tekton</a> folder includes the items related with Tekton and Pipelines.</li> <li> <a href="//./e2e-test/README.md">e2e-test</a> folder includes the end-to-end test suites of the product.</li> </ul> <h2> Trunk-based Development </h2> <p>A Monorepo is a specific <a href="https://trunkbaseddevelopment.com/">Trunk-Based Development</a> implementation where the product team puts its source for all applications/services/libraries/frameworks into one repository and forces team members to commit together in that trunk - atomically.</p> <p>The <code>trunk</code> branch is defined basically as productive and any change merged there is candidate to be deployed in any environment, including production. There are not any other starting point to promote changes, as everything is integrated in the trunk. Other branches are considered ephemeral, to manage short pieces of changes, with a review process (by <a href="https://openpracticelibrary.com/practice/pair-programming/">pair programming</a> or by a Pull Request) before to be merged into the trunk.</p> <p>This method allows the team members to develop fast small chunks (many times as <a href="https://openpracticelibrary.com/practice/feature-toggles/">features flags</a>or not), with sooner integrations cycles, reducing merge conflicts, and promoting changes to others faster.</p> <h2> GitOps </h2> <p><a href="https://openpracticelibrary.com/practice/gitops/">GitOps</a> defines the source of truth a Git repository, where everything starts from there and defining the desired state of our product.</p> <h2> Cloud Native Application </h2> <p>Designing our product as a <a href="https://en.wikipedia.org/wiki/Cloud_native_computing">Cloud Native</a> solution will bring us many benefits in cloud environments. Most of the cases a Microservice architecture following the <a href="https://12factor.net/">Twelve-Factor App</a>methodology is the right starting point. Our product will implement that methodology.</p> <h2> Continuous Integration, Continuous Delivery and Continuous Deployment </h2> <p>Any product in the new automated era must be done without the most well-known benefits of<a href="https://openpracticelibrary.com/practice/continuous-integration/">Continuous Integration</a>,<a href="https://openpracticelibrary.com/practice/continuous-delivery/">Continuous Delivery</a> and<a href="https://openpracticelibrary.com/practice/continuous-deployment/">Continuous Deployment</a>. Otherwise you are failing from the beginning.</p> <h2> Sealed Secrets </h2> <p>GitOps means <strong>โ€œif itโ€™s not in Git, itโ€™s NOT REALโ€</strong> , so it is a challenge to store sensitive data, like credentials, in Git repositories, where many people can access?. OpenShift provides a good way to manage sensitive data in the platform, but we need to extend it with other great tools to store sensitive data in Git without any break of security.</p> <p>Here <a href="https://github.com/bitnami-labs/sealed-secrets">Sealed Secrets</a> arrives to help us.</p> <h2> Playing with our Product Monorepo </h2> <p>Now, it is time to play ๐ŸŽฒ.</p> <p>This repository defines a sample product with the following applications:</p> <ul> <li>A sample Angular application as frontend for the final users. Details <a href="//./apps/sample-frontend/README.md">here</a> </li> <li>A sample Quarkus application as backend to manage the <em>business logic</em> of our product. Details <a href="//./apps/sample-backend/README.md">here</a> </li> </ul> <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s--eM3TrOfB--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://blog.jromanmartin.io/2022/03/25/img/product-deployment-topology.png" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--eM3TrOfB--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://blog.jromanmartin.io/2022/03/25/img/product-deployment-topology.png" alt="Product Monorepo Topology" width="" height=""></a></p> <h3> Requirements </h3> <p>This repository had been developed and tested in the following environment:</p> <ul> <li>Red Hat OpenShift Container Platform 4.10</li> <li>Red Hat OpenShift GitOps 1.4.3 (ArgoCD)</li> <li>Red Hat OpenShift Pipelines 1.6.2 (Tekton)</li> <li>Sealed Secrets Helm Chart 1.16.1</li> </ul> <h3> Bootstrapping Red Hat OpenShift Container Platform </h3> <p>To prepare your OCP environment, review and follow the <a href="//./bootstrap/README.md">bootstrap instructions</a></p> <p>If everything goes fine, your environment should look like as:</p> <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s--YRLWqwFE--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://blog.jromanmartin.io/2022/03/25/img/cicd-tools-deployment-topology.png" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--YRLWqwFE--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://blog.jromanmartin.io/2022/03/25/img/cicd-tools-deployment-topology.png" alt="CICD Tools Deployment Topology" width="" height=""></a></p> <h3> GitOps with ArgoCD </h3> <p>To prepare the GitOps scenario with ArgoCD, review and follow the <a href="//./argocd/README.md">instructions</a>.</p> <p>If everything goes fine, your ArgoCD should look like as:</p> <p><a href="https://res.cloudinary.com/practicaldev/image/fetch/s--xvSvh6sz--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://blog.jromanmartin.io/2022/03/25/img/argocd-deployment-topology.png" class="article-body-image-wrapper"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--xvSvh6sz--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://blog.jromanmartin.io/2022/03/25/img/argocd-deployment-topology.png" alt="ArgoCD Deployment Topology" width="" height=""></a></p> <h3> CICD with Tekton Pipelines </h3> <p>This Product Monorepo has a set of different pipelines to cover the Software Delivery Life cycle, integrated in our GitOps approach. The pipelines are described <a href="//./tekton/README.md">here</a>.</p> <h2> Feedback, Comments, and improvements </h2> <p>As this is an <em>opinionated</em> approach, from my field experience in real scenarios and use cases, I am always open to learn from other experiences and use cases. Feel free to comment, improve or change my mind with your great ideas. Donโ€™t forget to review our <a href="//./CONTRIBUTING.md">Contribution Guide</a> do it in many different ways (issues, pull-request, comments, โ€ฆ), donโ€™t miss the chance to do it.</p> <p>I also open to share this approach, techniques and tools in community, meetup or simple group of colleagues around topics such as DevOps, Agile, GitOps, Cloud Native, โ€ฆ If you think that I can participate, please, let me know it.</p> <p>If you are here, thank you so much. ๐Ÿ˜„ ๐ŸŽ‰</p> howto quarkus springboot Lessons learned migrating Spring Boot to Quarkus Jose Roman Martin Gil Fri, 03 Dec 2021 09:15:00 +0000 https://dev.to/rmarting/lessons-learned-migrating-spring-boot-to-quarkus-1ccb https://dev.to/rmarting/lessons-learned-migrating-spring-boot-to-quarkus-1ccb <p>This blog post describes a set of lessons learned from my personal experience migrating Spring Boot applications to Quarkus. The article does not cover all the topics, approaches, architectures or designs to keep in mind for an enterprise full migration project, but it includes a set of conclusions from a personal perspective.</p> <p>Cloud Native Applications, Microservices Architectures, Event-Driven Architectures, Serverless, โ€ฆ are the most common patterns, designs and topics used by Enterprise, Start-ups, and Software Companies to design and deploy new applications in this new Cloud Era (a.k.a <a href="https://www.infoq.com/articles/microservices-post-kubernetes/">Kubernetes Era</a>). To build these kinds of new applications exists a list of different technologies, frameworks and languages (Go, Node.JS, Java, โ€ฆ) however one of the most extended and used is Spring Boot.</p> <p>Spring Boot is a well-known framework, with a large community of developers, long history and friendly for many developers. However Spring Boot has other behaviors that could not fit well in a Cloud Native environment (resources consumption, startup time, response time, development lifecycle, โ€ฆ).</p> <p><a href="https://quarkus.io/">Quarkus</a> is the new player in the playground to design new applications under these paradigms.</p> <p>Quarkus is a full-stack, Kubernetes-native Java framework made for Java Virtual Machines and native compilation. Quarkus is crafted from best-of-breed Java libraries and standards with amazingly fast boot times and incredibly low memory in container orchestration platforms like Kubernetes.</p> <p>Quarkus has a clear vision based in:</p> <ul> <li> <a href="https://quarkus.io/container-first">Container First</a>: Optimized for low memory usage and fast startup times.</li> <li> <a href="https://quarkus.io/continuum">Imperative and Reactive</a>: Designed with this new world in mind and provides first-class support for these different paradigms.</li> <li> <a href="https://quarkus.io/developer-joy">Developer Joy</a>: Designed to make happy and fun the developerโ€™s life.</li> <li> <a href="https://quarkus.io/standards">Community and Standards</a>: No need to learn new technologies, designed on top of proven standards (Eclipse MicroProfile, JAX-RS, JPA, โ€ฆ)</li> <li> <a href="https://quarkus.io/kubernetes-native">Kube-native</a>: Providing tools optimized for Kubernetes.</li> </ul> <p><strong>Houston, we have a problem!</strong> Quarkus versus Spring Boot? Quarkus? Spring Boot? Which one?</p> <p>In many migration projects (frameworks, application servers, jdk, โ€ฆ) the effort to adapt the source code to the new platform was one key. Refactoring code implies a range effort between trivial to epic and it could decline the decision to go or not to go. Refactoring from Spring Boot to Quarkus could be another one.</p> <p>This article describes the following migration approaches from Spring Boot to Quarkus:</p> <ul> <li>Migrating to Quarkus Extensions for Spring Boot</li> <li>Refactoring to Standard Libraries and Quarkus Extensions</li> </ul> <p>๐Ÿšจ๐Ÿšจ <strong>Disclaimer and Spoiler Alert</strong> This article is not an official migration guide. ๐Ÿšจ๐Ÿšจ</p> <p>Any kind of migration requires analyzing different things to answer questions such as why?, who?, when?, how?, where?. These questions are not easy to analyze and describe in a single article because they involve a large number of aspects like: processes, people, management, testing, โ€ฆ but who does not listen to sentences such as: <em>Java is dead now</em>; <em>XX framework is better for cloud containers</em>, <em>Java consumes a lot of resources</em>, โ€ฆ Well, <strong>Quarkus changes many of those sentences</strong>.</p> <p>This article only wants to focus on some aspects of how to code/refactor the source code from Spring Boot to Quarkus. It does not cover all the features or capabilities of both frameworks but it summarizes a set of lessons learned from my personal experience. Quarkus is a highly active community so some of these lessons could change in the future (maybe in a few days).</p> <h2> Application to migrate </h2> <p>This blog post is based on a Spring Boot application with the following modules or components:</p> <ul> <li>Spring Boot 2</li> <li>REST Endpoints based in Spring Web</li> <li>Apache Kafka as messaging system</li> <li>Apicurio Service Registry as API schema registry</li> <li>Avro schemas</li> </ul> <p>It is a base to start to analyze this migration, where some other common features are not included (e.g: Persistence in Databases) to reduce the scope of this migration.</p> <p>The original code of this application is available <a href="https://github.com/rmarting/kafka-clients-sb-sample">here</a>.</p> <p>The original application starts in 5 seconds:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>2021-12-03 09:33:56.724 INFO 1 --- [main] com.rmarting.kafka.Application : Started Application in 5.132 seconds (JVM running for 5.76) </code></pre> </div> <h2> Migrating to Quarkus Extensions for Spring Boot </h2> <p>This approach is focused on reducing the number of changes and reuse as much code as possible. This approach is available thanks to a set of Quarkus Extensions for Spring Boot, designed to provide a compatibility layer for Spring Boot. At the time of writing this article the following Quarkus Extensions for Spring are available:</p> <ul> <li> <strong>spring-di</strong> : Compatibility layer for Spring dependency injection.</li> <li> <strong>spring-web</strong> : Compatibility layer for Spring Web.</li> <li> <strong>spring-boot-properties</strong> : Compatibility layer to set up your Spring Boot using @ConfigurationProperties annotations.</li> <li> <strong>spring-security</strong> : Compatibility layer for Spring Security.</li> <li> <strong>spring-cache</strong> : Compatibility layer for Spring Cache annotations.</li> <li> <strong>spring-data-jpa</strong> : Compatibility layer for Spring Data JPA repositories.</li> <li> <strong>spring-scheduled</strong> : Compatibility layer for Spring Scheduled.</li> <li> <strong>spring-cloud-config-client</strong> : Compatibility layer to read configuration properties at runtime from the Spring Cloud Config Server.</li> </ul> <p><strong>NOTE</strong> : Some extensions are considered preview, so backward compatibility and presence in the ecosystem is not guaranteed.</p> <p>The main list of changes done to migrate the application is:</p> <ul> <li>Quarkus requires JDK 11.</li> <li> <code>spring-di</code>, <code>spring-web</code> and <code>spring-boot-properties</code> extensions are basically <em>mandatory</em> for any Spring Boot applications. We could say that they are the base for any migration to Quarkus Spring.</li> <li>These extensions provide the base features of Spring Boot such as: Dependency Injection, Web, Configuration.</li> <li>Although the compatibility layer supports most of the Spring DI capabilities, some arcane features may not be supported.</li> <li>Spring Web Annotations could be maintained exactly equals.</li> <li>Swagger annotations must be refactored to use MicroProfile OpenAPI. This refactor basically needs to use classes from<code>org.eclipse.microprofile.openapi.annotations</code> package.</li> <li>OpenAPI and Swagger capabilities are provided now by the<code>quarkus-smallrye-openapi</code> extension, not needed by other OpenAPI or Swagger dependencies.</li> <li>Health checks (actuators provided by <code>spring-boot-starter-actuator</code>) are now provided by <code>quarkus-smallrye-health</code> extensions. It requires new liveness and readiness proves in your deployment in Kubernetes.</li> <li>Integration with Kafka using the Kafka Producer and Consumer API (provided by the Kafka Clients) only requires the use of <code>quarkus-kafka-client</code> extension.</li> <li>Spring Kafka has not an equivalent compatible Spring extension, however Quarkus provides <code>quarkus-smallrye-reactive-messaging-kafka</code> extension with a set of new annotations to consume, produce or data streaming with Apache Kafka. It means a small set of changes.</li> </ul> <p>The result of this migration is available <a href="https://github.com/rmarting/kafka-clients-sb-sample/tree/feature/quarkus-edition">here</a>.</p> <p>This new application starts in less of 2 seconds:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Dec 03, 2021 9:51:17 AM io.quarkus.bootstrap.runner.Timing printStartupTime INFO: kafka-clients-sb-sample 3.0.0-SNAPSHOT on JVM (powered by Quarkus 1.13.7.Final) started in 1.887s. Listening on: http://0.0.0.0:8181 </code></pre> </div> <h2> Refactoring to Standard Libraries and Quarkus Extensions </h2> <p>This approach implies a change in the mindset of your application, aligning with standards and refactoring your code. However you will get a full-compliant Quarkus application and then you are able to use all its empowerment.</p> <p>For this approach I need to map the Spring Boot features with the right Quarkus Extensions:</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Spring Boot Feature</th> <th>Quarkus Extension</th> </tr> </thead> <tbody> <tr> <td>spring-di</td> <td><a href="https://quarkus.io/guides/cdi">Quarkus CDI</a></td> </tr> <tr> <td>spring-boot-starter-web</td> <td><a href="https://quarkus.io/guides/rest-json">JAX-RS Services</a></td> </tr> <tr> <td>spring-boot-starter-actuator</td> <td><a href="https://quarkus.io/guides/smallrye-health">MicroProfile Health</a></td> </tr> <tr> <td>springdoc-openapi-ui</td> <td><a href="https://quarkus.io/guides/openapi-swaggerui">OpenAPI and Swagger UI</a></td> </tr> <tr> <td>spring-kafka</td> <td><a href="https://quarkus.io/guides/kafka">Kafka with Reactive Messaging</a></td> </tr> </tbody> </table></div> <p>The main list of changes done to migrate the application is:</p> <ul> <li>Quarkus requires JDK 11.</li> <li>Refactor <code>@Service</code>, <code>@Component</code>, <code>@Autowired</code> Spring annotations to <code>@Singleton</code>, <code>@ApplicationScoped</code>, <code>@Inject</code> CDI annotations.</li> <li>Refactor <code>@Configuration</code>, <code>@Value</code> Spring annotations to<code>@ApplicationScoped</code>, <code>@ConfigProperty</code> Quarkus annotations</li> <li>Refactor Spring Web Annotations to JAX-RS Annotations. This guide includes a <a href="https://quarkus.io/guides/spring-web#conversion-table">conversion table</a>.</li> <li>Swagger annotations must be refactored to use MicroProfile OpenAPI. This refactor basically needs to use classes from<code>org.eclipse.microprofile.openapi.annotations</code> package.</li> <li>OpenAPI and Swagger capabilities are provided now by <code>quarkus-smallrye-openapi</code> extension, not needed by other OpenAPI or Swagger dependencies.</li> <li>Health checks (actuators provided by <code>spring-boot-starter-actuator</code>) are now provided by <code>quarkus-smallrye-health</code> extension. It requires new liveness and readiness proves in your deployment in Kubernetes.</li> <li>Integration with Kafka using the Kafka Producer and Consumer API (provided by the Kafka Clients) only requires the use of <code>quarkus-kafka-client</code> extension.</li> <li>Spring Kafka has not an equivalent compatible Spring extension, however Quarkus provides <code>quarkus-smallrye-reactive-messaging-kafka</code> extension with a set of new annotations to consume, produce or data streaming with Apache Kafka. It means a small set of changes.</li> </ul> <p>The result of this refactoring is available <a href="https://github.com/rmarting/kafka-clients-quarkus-sample">here</a>.</p> <p>This application starts in 1.4 seconds:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>Dec 03, 2021 10:21:17 AM io.quarkus.bootstrap.runner.Timing printStartupTime INFO: kafka-clients-sb-sample 3.0.0-SNAPSHOT on JVM (powered by Quarkus 1.13.7.Final) started in 1.387s. Listening on: http://0.0.0.0:8181 </code></pre> </div> <h2> Lessons Learned </h2> <p>Both migration experiences could be summarized in the next lessons learned:</p> <ul> <li>Migration to Quarkus has been feasible with less effort migration. Both approaches did not require a full investment or hard work.</li> <li>Quarkus Extensions for Spring Boot are enough for most common modules (<code>di</code>, <code>web</code>, <code>jpa</code>, <code>security</code>, โ€ฆ) but maybe not cover yours (<code>jta</code>, <code>web-services</code>, complex injection references, โ€ฆ). Analyzing the application is <strong>mandatory</strong> to get the gap.</li> <li>Quarkus Extensions for Spring Boot could not cover all Spring features, then a refactor to Quarkus could be needed (Health Endpoints, OpenAPI, Swagger UI, Messaging Integration). However, the effort to refactor it was not so hard.</li> <li>Refactoring to Quarkus involves moving your code to standard libraries (that hopefully you already know like for instance JAX-RS), so the learning curve to start with Quarkus is minimal.</li> <li>Refactor to Quarkus will give you the full-power of Quarkus and its extensions.</li> <li>Bugs and issues could appear in both migrations. Quarkus is stable and it is growing up so fast, resolving them and adding new features. You can check it in the <a href="https://github.com/quarkusio/quarkus/releases">releases page</a>.</li> <li>Full Quarkus migrated application was the faster one after completing the refactoring, as you can see in the following chart. It is not a complete performance test, but it might give you an idea about the performance capabilities of Quarkus:</li> </ul> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>Implementation</th> <th>Startup</th> </tr> </thead> <tbody> <tr> <td>Spring Boot</td> <td>5 seconds</td> </tr> <tr> <td>Quarkus extensions for Spring</td> <td>1.7 seconds ๐Ÿš€</td> </tr> <tr> <td>Quarkus</td> <td>1.4 seconds ๐Ÿš€</td> </tr> <tr> <td>Quarkus Native</td> <td>0.042 seconds ๐Ÿš€๐Ÿš€</td> </tr> </tbody> </table></div> <p>There is a list of other components not covered (e.g: testing, JPA, Cloud integration, security, โ€ฆ) in this blog post because it will extend the scope and length of this article. Quarkus is a highly active community where every day new features, issues and extensions are moving so some of these lessons learned could be covered, resolved or fixed soon.</p> <h2> Getting starting my migration </h2> <p>Migrating Spring Boot to Quarkus requires an effort to identify the best approach from the current state (AS-IS) to the final state (TO-BE) . There is not a silver bullet to migrate applications but there are some tools and references that could help:</p> <ul> <li> <a href="https://developers.redhat.com/products/mta/overview">Red Hat Migration Toolkit for Applications</a>: This tool could analyze your code to identify the main migration issues. The latest version includes a set of rules to check your code and identify the main issues to migrate to Quarkus.</li> <li> <a href="https://quarkus.io/guides/">Quarkus Guides</a> are a great resource for getting started in Quarkus.</li> <li> <a href="https://github.com/quarkusio/quarkus-quickstarts">Quarkus QuickStarts</a> is a large repository of code with many samples.</li> <li> <a href="https://quarkus.io/blog/">Quarkus Blog</a>.</li> <li> <a href="https://dzone.com/articles/migrating-a-spring-boot-application-to-quarkus-cha">Migrating SpringBoot PetClinic REST to Quarkus</a> by Jonathan Vila as another migration reference from Spring Boot.</li> <li><a href="https://developers.redhat.com/blog/2020/04/10/migrating-a-spring-boot-microservices-application-to-quarkus">Migrating a Spring Boot microservices application to Quarkus</a></li> <li><a href="https://developers.redhat.com/blog/2020/07/17/migrating-spring-boot-tests-to-quarkus">Migrating Spring Boot tests to Quarkus</a></li> </ul> <p>๐ŸŽ‰ Enjoy your journey to Quarkus. ๐ŸŽ‰</p> howto quarkus springboot migration Migrating Kafka clusters with MirrorMaker2 and Strimzi Jose Roman Martin Gil Fri, 19 Nov 2021 00:00:00 +0000 https://dev.to/rmarting/migrating-kafka-clusters-with-mirrormaker2-and-strimzi-577h https://dev.to/rmarting/migrating-kafka-clusters-with-mirrormaker2-and-strimzi-577h <p>In this article we (my colleague <a href="https://www.linkedin.com/in/manuel-schindler-aa0397118/">Manuel Schindler</a><br> and I) would like to focus on the most common challenges to migrate Apache<br> Kafka clusters between different OpenShift platforms, and how to overcome<br> them by using Apache MirrorMaker and <a href="https://strimzi.io/">Strimzi Operators</a>.</p> <p><strong>๐Ÿ—ž๏ธ โ„น๏ธ LATEST NEWS โ„น๏ธ ๐Ÿ—ž๏ธ</strong> This article has<br> been accepted and published in the <a href="https://strimzi.io/blog/2021/11/22/migrating-kafka-with-mirror-maker2/">Strimzi Blog</a><br> community ๐ŸŽ‰ <a href="">here</a>. We are glad ๐Ÿคฉ to help and contribute in the Strimzi Community with our<br> experience and knowledge of this amazing Open Source project.</p> <p><strong>Thank you so much Strimzi Community</strong>!!! ๐Ÿ’ช ๐ŸŽ‰</p> <h3> Greetings </h3> <p>Many thanks to some great colleagues for help us reviewing this content:</p> <ul> <li>Hugo Guerrero <a href="https://twitter.com/hguerreroo"></a> <a href="https://www.linkedin.com/in/hugoguerrero/"></a> </li> <li>Jakub Scholz <a href="https://twitter.com/scholzj"></a> <a href="https://www.linkedin.com/in/scholzj/"></a> </li> <li>Rafael Yanez <a href="https://www.linkedin.com/in/ryanezillescas/"></a> </li> <li>Paolo Patierno <a href="https://twitter.com/ppatierno"></a> <a href="https://www.linkedin.com/in/paolopatierno/"></a> </li> </ul> strimzi mirrormaker2 migration apachekafka Integrating Quarkus with Apicurio Service Registry Jose Roman Martin Gil Fri, 18 Dec 2020 09:15:00 +0000 https://dev.to/rmarting/integrating-quarkus-with-apicurio-service-registry-196i https://dev.to/rmarting/integrating-quarkus-with-apicurio-service-registry-196i <p>Step by step most of the new cloud native applications and microservices designs are based in<a href="https://developers.redhat.com/topics/event-driven" rel="noopener noreferrer">event-driven architecture (EDA)</a> to respond to real-time information based on sending and receiving of information about individual events. This kind of architecture is based on asynchronous non-blocking communication between event producers and consumers through a event streaming backbone, such as Apache Kafka running on top of Kubernetes. In these scenarios, where a large number of different events are managed, is very important to define a governance model where each event could be defined as an API to allow producers and consumers to produce and consume checked and validated events. A Service Registry will help us.</p> <p>From my field experience with many projects I found that the most typical landscape is based in the most well-know next components:</p> <ul> <li> <a href="https://strimzi.io/" rel="noopener noreferrer">Strimzi</a> to deploy Apache Kafka clusters as streaming backbone.</li> <li> <a href="https://www.apicur.io/" rel="noopener noreferrer">Apicurio Service Registry</a> as datastore for events API.</li> <li> <a href="https://www.openshift.com/" rel="noopener noreferrer">OpenShift Container Platform</a> to deploy and run the different components.</li> <li> <a href="https://quarkus.io/" rel="noopener noreferrer">Quarkus</a> as framework to develop client applications.</li> <li> <a href="https://avro.apache.org/" rel="noopener noreferrer">Avro</a> as data serialization system to declare schemas as events API.</li> </ul> <p>This article describes how easy is to integrate your Quarkus applications with Apicurio Service Registry.</p> <h2> ๐Ÿค– Apicurio Service Registry </h2> <p>Service Registry is a datastore for sharing standard event schemas and API designs across API and event-driven architectures. Service Registry decouples the structure of your data from your client applications, and to share and manage your data types and API descriptions at runtime. Decouples your data structure from your client applications reduces costs by decreasing overall message size, creates efficiencies by increasing consistent reuse of schemas and API designs across your organization.</p> <p>Some of the most common uses cases where Service Registry helps us are:</p> <ul> <li>Client applications can dynamically push or pull the latest schema updates to or from Service Registry at runtime without needing to redeploy.</li> <li>Developer teams can query the registry for existing schemas required for services already deployed in production.</li> <li>Developer teams can register new schemas required for new services in development or rolling to production.</li> <li>Store schemas used to serialize and deserialize messages, which can then be referenced from your client applications to ensure that the messages that they send and receive are compatible with those schemas.</li> </ul> <p>Apicurio is a open source project that provides a Service Registry ready to be involved in this scenario with the following main features:</p> <ul> <li>Support for multiple payload formats for standard event schemas and API specifications.</li> <li>Pluggable storage options including AMQ Streams, embedded Infinispan, or PostgreSQL database.</li> <li>Registry content management using a web console, REST API command, Maven plug-in, or Java client.</li> <li>Rules for content validation and version compatibility to govern how registry content evolves over time.</li> <li>Full Apache Kafka schema registry support, including integration with Kafka Connect for external systems.</li> <li>Client serializer/deserializer (Serdes) to validate Kafka and other message types at runtime.</li> <li>Cloud-native Quarkus Java runtime for low memory footprint and fast deployment times.</li> <li>Compatibility with existing Confluent schema registry client applications.</li> <li>Operator-based installation of Service Registry on OpenShift.</li> </ul> <h2> ๐Ÿ’ป Client Applications Workflow </h2> <p>The typical workflow when we introduce a Service Registry in our architecture is:</p> <ul> <li>Declare the event schema using some of the most data formats like Apache Avro, JSON Schema, Google Protocol Buffers, OpenAPI, AsyncAPI, GraphQL, Kafka Connect schemas, WSDL or XML schemas (xsd).</li> <li>Register the schema as artifact in Service Registry through the Service Registry UI, API REST, Maven PlugIn or Java clients. From there client applications can then use that schema to validate that messages conform to the correct data structure at runtime.</li> <li>Kafka Producer applications use a serializer to encode messages that conform to a specific event schema.</li> <li>Kafka Consumer applications then use a deserializer to validate that messages have been serialized using that correct schema, based on a specific schema ID.</li> </ul> <p>This workflow ensures consistent schema use and helps to prevent data errors at runtime.</p> <h2> ๐Ÿ“„ Avro Schemas into Service Registry </h2> <p>Avro provides a <a href="https://avro.apache.org/docs/current/spec.html#schemas" rel="noopener noreferrer">JSON schema specification</a> to declare a large variety of data structures, such as our simple example:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight json"><code><span class="p">{</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Message"</span><span class="p">,</span><span class="w"> </span><span class="nl">"namespace"</span><span class="p">:</span><span class="w"> </span><span class="s2">"io.jromanmartin.kafka.schema.avro"</span><span class="p">,</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"record"</span><span class="p">,</span><span class="w"> </span><span class="nl">"doc"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Schema for a Message."</span><span class="p">,</span><span class="w"> </span><span class="nl">"fields"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"timestamp"</span><span class="p">,</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"long"</span><span class="p">,</span><span class="w"> </span><span class="nl">"doc"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Message timestamp."</span><span class="w"> </span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w"> </span><span class="nl">"name"</span><span class="p">:</span><span class="w"> </span><span class="s2">"content"</span><span class="p">,</span><span class="w"> </span><span class="nl">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"string"</span><span class="p">,</span><span class="w"> </span><span class="nl">"doc"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Message content."</span><span class="w"> </span><span class="p">}</span><span class="w"> </span><span class="p">]</span><span class="w"> </span><span class="p">}</span><span class="w"> </span></code></pre> </div> <p>This schema will define a simple message event.</p> <p>Avro also provides a <a href="https://avro.apache.org/docs/current/gettingstartedjava.html" rel="noopener noreferrer">Maven Plugin</a> to autogenerate Java classes based in the schema definitions (<code>.avsc</code> files)</p> <p>Now we could publish it into Service Registry to be used in runtime for our client applications. Apicurio Maven Plugin is an easy way to publish the schemas into the Service Registry with a simple definition in our <code>pom.xml</code> file.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight xml"><code><span class="nt">&lt;plugin&gt;</span> <span class="nt">&lt;groupId&gt;</span>io.apicurio<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>apicurio-registry-maven-plugin<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>${apicurio.version}<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;executions&gt;</span> <span class="nt">&lt;execution&gt;</span> <span class="nt">&lt;phase&gt;</span>generate-sources<span class="nt">&lt;/phase&gt;</span> <span class="nt">&lt;goals&gt;</span> <span class="nt">&lt;goal&gt;</span>register<span class="nt">&lt;/goal&gt;</span> <span class="nt">&lt;/goals&gt;</span> <span class="nt">&lt;configuration&gt;</span> <span class="nt">&lt;registryUrl&gt;</span>${apicurio.registry.url}<span class="nt">&lt;/registryUrl&gt;</span> <span class="nt">&lt;artifactType&gt;</span>AVRO<span class="nt">&lt;/artifactType&gt;</span> <span class="nt">&lt;artifacts&gt;</span> <span class="c">&lt;!-- Schema definition for TopicIdStrategy strategy --&gt;</span> <span class="nt">&lt;messages-value&gt;</span>${project.basedir}/src/main/resources/schemas/message.avsc<span class="nt">&lt;/messages-value&gt;</span> <span class="nt">&lt;/artifacts&gt;</span> <span class="nt">&lt;/configuration&gt;</span> <span class="nt">&lt;/execution&gt;</span> <span class="nt">&lt;/executions&gt;</span> <span class="nt">&lt;/plugin&gt;</span> </code></pre> </div> <p>Using Apicurio Maven Plugin in the our application Maven Lifecycle could help us to define or extend our ALM (also our CI/CD pipelines) to publish or update the schemas every time we released new versions of them. It is not an objective of this article, but something that you could analyze more.</p> <p>As soon we published our schema into Service Registry we could manage from the UI as:</p> <p><a href="http://blog.jromanmartin.io/images/apicurio-registry/apicurio-artifact-details.png" rel="noopener noreferrer"><img src="https://media.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/http%3A%2F%2Fblog.jromanmartin.io%2Fimages%2Fapicurio-registry%2Fapicurio-artifact-details.png" title="Apicurio Service Registry Artifact Details"></a></p> <h2> ๐Ÿš€ Quarkus, Apache Kafka and Service Registry </h2> <p>Quarkus provides a set of dependencies to allow in our application to produce and consume messages to and from Apache Kafka. It is very straight forward to use it onces we add the dependency in our <code>pom.xml</code> file:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight xml"><code><span class="c">&lt;!-- Kafka Clients --&gt;</span> <span class="nt">&lt;dependency&gt;</span> <span class="nt">&lt;groupId&gt;</span>io.quarkus<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>quarkus-kafka-client<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;/dependency&gt;</span> <span class="c">&lt;!-- Reactive Messaging --&gt;</span> <span class="nt">&lt;dependency&gt;</span> <span class="nt">&lt;groupId&gt;</span>io.quarkus<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>quarkus-smallrye-reactive-messaging-kafka<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;/dependency&gt;</span> </code></pre> </div> <p>Connect your application with the Apache Kafka cluster is so easy with the following property in your <code>application.properties</code> file:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>kafka.bootstrap.servers = my-kafka-kafka-bootstrap:9092 </code></pre> </div> <p>Apicurio Service Registry provides Kafka client serializer/deserializer for Kafka producer and consumer applications. To add them into our application, you must add the next dependency:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight xml"><code><span class="c">&lt;!-- Apicurio Serializer/Deserializer --&gt;</span> <span class="nt">&lt;dependency&gt;</span> <span class="nt">&lt;groupId&gt;</span>io.apicurio<span class="nt">&lt;/groupId&gt;</span> <span class="nt">&lt;artifactId&gt;</span>apicurio-registry-utils-serde<span class="nt">&lt;/artifactId&gt;</span> <span class="nt">&lt;version&gt;</span>${apicurio.version}<span class="nt">&lt;/version&gt;</span> <span class="nt">&lt;/dependency&gt;</span> </code></pre> </div> <h3> ๐Ÿ“ฅ Producing Messages from Quarkus </h3> <p>Quarkus provides a set of properties and beans to declare Kafka Producers to send messages (in our case Avro-schema instances) to Apache Kafka. The most important properties to set up are:</p> <ul> <li> <code>key.serializer</code>: Identifies the serializer class to serialize the key of the Kafka record.</li> <li> <code>value.serializer</code>: Identifies the serializer class to serialize the value of the Kafka record.</li> </ul> <p>Here we have to add some specific values in these properties to allow the serialization process using Avro schemas registered in the Service Registry. Basically we need to identify the following concepts:</p> <ul> <li>Serializer class to use Avro schemas, this class is provider by the Apicurio SerDe class: <code>io.apicurio.registry.utils.serde.AvroKafkaSerializer</code> </li> <li>Apicurio Service registry endpoint to validate schemas: <code>apicurio.registry.url</code> </li> <li>Apicurio Service strategy to look up the schema definition: <code>apicurio.registry.artifact-id</code> </li> </ul> <p>So a sample definition for a producer bean to send messages could be similar to:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight java"><code> <span class="nd">@Produces</span> <span class="nd">@RequestScoped</span> <span class="kd">public</span> <span class="nc">Producer</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">,</span> <span class="nc">Message</span><span class="o">&gt;</span> <span class="nf">createProducer</span><span class="o">()</span> <span class="o">{</span> <span class="nc">Properties</span> <span class="n">props</span> <span class="o">=</span> <span class="k">new</span> <span class="nc">Properties</span><span class="o">();</span> <span class="c1">// Kafka Bootstrap</span> <span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">ProducerConfig</span><span class="o">.</span><span class="na">BOOTSTRAP_SERVERS_CONFIG</span><span class="o">,</span> <span class="n">kafkaBrokers</span><span class="o">);</span> <span class="c1">// Security</span> <span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">AdminClientConfig</span><span class="o">.</span><span class="na">SECURITY_PROTOCOL_CONFIG</span><span class="o">,</span> <span class="n">kafkaSecurityProtocol</span><span class="o">);</span> <span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">SaslConfigs</span><span class="o">.</span><span class="na">SASL_MECHANISM</span><span class="o">,</span> <span class="s">"SCRAM-SHA-512"</span><span class="o">);</span> <span class="n">props</span><span class="o">.</span><span class="na">put</span><span class="o">(</span><span class="nc">SaslConfigs</span><span class="o">.</span><span class="na">SASL_JAAS_CONFIG</span><span class="o">,</span> <span class="s">"org.apache.kafka.common.security.scram.ScramLoginModule required username=\""</span> <span class="o">+</span> <span class="n">kafkaUser</span> <span class="o">+</span> <span class="s">"\" password=\""</span> <span class="o">+</span> <span class="n">kafkaPassword</span> <span class="o">+</span> <span class="s">"\";"</span><span class="o">);</span> <span class="c1">// Producer Client</span> <span class="n">props</span><span class="o">.</span><span class="na">putIfAbsent</span><span class="o">(</span><span class="nc">ProducerConfig</span><span class="o">.</span><span class="na">CLIENT_ID_CONFIG</span><span class="o">,</span> <span class="n">producerClientId</span> <span class="o">+</span> <span class="s">"-"</span> <span class="o">+</span> <span class="n">getHostname</span><span class="o">());</span> <span class="c1">// Serializer for Keys and Values props.putIfAbsent(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());</span> <span class="n">props</span><span class="o">.</span><span class="na">putIfAbsent</span><span class="o">(</span><span class="nc">ProducerConfig</span><span class="o">.</span><span class="na">VALUE_SERIALIZER_CLASS_CONFIG</span><span class="o">,</span> <span class="nc">AvroKafkaSerializer</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getName</span><span class="o">());</span> <span class="c1">// Service Registry props.putIfAbsent(AbstractKafkaSerDe.REGISTRY_URL_CONFIG_PARAM, serviceRegistryUrl);</span> <span class="c1">// Topic Id Strategy (schema = topicName-(key|value)) - Default Strategy</span> <span class="n">props</span><span class="o">.</span><span class="na">putIfAbsent</span><span class="o">(</span><span class="nc">AbstractKafkaSerializer</span><span class="o">.</span><span class="na">REGISTRY_ARTIFACT_ID_STRATEGY_CONFIG_PARAM</span><span class="o">,</span> <span class="nc">TopicIdStrategy</span><span class="o">.</span><span class="na">class</span><span class="o">.</span><span class="na">getName</span><span class="o">());</span> <span class="c1">// Acknowledgement</span> <span class="n">props</span><span class="o">.</span><span class="na">putIfAbsent</span><span class="o">(</span><span class="nc">ProducerConfig</span><span class="o">.</span><span class="na">ACKS_CONFIG</span><span class="o">,</span> <span class="n">acks</span><span class="o">);</span> <span class="k">return</span> <span class="k">new</span> <span class="nc">KafkaProducer</span><span class="o">&lt;&gt;(</span><span class="n">props</span><span class="o">);</span> <span class="o">}</span> </code></pre> </div> <p>And you could finally send Messages (storing the artifact ID from Service Registry) to Apache Kafka:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight java"><code> <span class="nd">@Inject</span> <span class="nc">Producer</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">,</span> <span class="nc">Message</span><span class="o">&gt;</span> <span class="n">producer</span><span class="o">;</span> <span class="kd">private</span> <span class="nc">MessageDTO</span> <span class="nf">publishRawMessage</span><span class="o">(</span><span class="kd">final</span> <span class="nd">@NotEmpty</span> <span class="nc">String</span> <span class="n">topicName</span><span class="o">,</span> <span class="kd">final</span> <span class="nd">@NotNull</span> <span class="nc">MessageDTO</span> <span class="n">messageDTO</span><span class="o">,</span> <span class="kd">final</span> <span class="kt">boolean</span> <span class="n">async</span><span class="o">)</span> <span class="o">{</span> <span class="c1">// ... </span> <span class="nc">RecordMetadata</span> <span class="n">metadata</span> <span class="o">=</span> <span class="n">producer</span><span class="o">.</span><span class="na">send</span><span class="o">(</span><span class="n">record</span><span class="o">).</span><span class="na">get</span><span class="o">();</span> <span class="c1">// ....</span> <span class="o">}</span> </code></pre> </div> <p>The message will be serialized adding the global ID associated to the schema used for this record. That global ID will be very important to consume later the message by the Kafka Consumer applications.</p> <p><strong>NOTE:</strong> This approach uses KafkaProducer API, however Quarkus includes the <code>Emitter</code> class to send messages easily. I developed my sample with the first approach to check that Kafka API is still valid using Quarkus.</p> <h3> ๐Ÿ“ค Consuming Messages from Quarkus </h3> <p>Quarkus also provides a set of properties and beans to declare Kafka Consumers to consume messages (in our case Avro-schema instances) from the Apache Kafka cluster. The most important properties to set up are:</p> <ul> <li> <code>key.deserializer</code>: Identifies the deserializer class to deserialize the key of the Kafka record.</li> <li> <code>value.deserializer</code>: Identifies the deserializer class to deserialize the value of the Kafka record.</li> </ul> <p>Here we have to add some specific values in these properties to allow the deserialization process using Avro schemas registered in the Service Registry. Basically we need to identify the following concepts:</p> <ul> <li>Deserializer class to use Avro schemas, this class is provider by the Apicurio SerDe class: <code>io.apicurio.registry.utils.serde.AvroKafkaDeserializer</code> </li> <li>Apicurio Service registry endpoint to get valid schemas: <code>apicurio.registry.url</code> </li> </ul> <p>So a sample configuration for a consumer template could be similar to:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code># Configure the Kafka source (we read from it) mp.messaging.incoming.messages.connector=smallrye-kafka mp.messaging.incoming.messages.group.id=${app.consumer.groupId}-mp-incoming-channel mp.messaging.incoming.messages.topic=messages.${app.bg.mode} mp.messaging.incoming.messages.key.deserializer=org.apache.kafka.common.serialization.StringDeserializer mp.messaging.incoming.messages.value.deserializer=io.apicurio.registry.utils.serde.AvroKafkaDeserializer mp.messaging.incoming.messages.properties.partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor mp.messaging.incoming.messages.apicurio.registry.url=${apicurio.registry.url} mp.messaging.incoming.messages.apicurio.registry.avro-datum-provider=io.apicurio.registry.utils.serde.avro.ReflectAvroDatumProvider </code></pre> </div> <p>You could declare a listener to consume messages (based in our Message schema) as:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight java"><code> <span class="nd">@Incoming</span><span class="o">(</span><span class="s">"messages"</span><span class="o">)</span> <span class="kd">public</span> <span class="nc">CompletionStage</span> <span class="nf">handleMessages</span><span class="o">(</span><span class="nc">Message</span> <span class="n">message</span><span class="o">)</span> <span class="o">{</span> <span class="nc">IncomingKafkaRecord</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">,</span> <span class="n">io</span><span class="o">.</span><span class="na">jromanmartin</span><span class="o">.</span><span class="na">kafka</span><span class="o">.</span><span class="na">schema</span><span class="o">.</span><span class="na">avro</span><span class="o">.</span><span class="na">Message</span><span class="o">&gt;</span> <span class="n">incomingKafkaRecord</span> <span class="o">=</span> <span class="o">(</span><span class="nc">IncomingKafkaRecord</span><span class="o">&lt;</span><span class="nc">String</span><span class="o">,</span> <span class="n">io</span><span class="o">.</span><span class="na">jromanmartin</span><span class="o">.</span><span class="na">kafka</span><span class="o">.</span><span class="na">schema</span><span class="o">.</span><span class="na">avro</span><span class="o">.</span><span class="na">Message</span><span class="o">&gt;)</span> <span class="n">message</span><span class="o">.</span><span class="na">unwrap</span><span class="o">(</span><span class="nc">IncomingKafkaRecord</span><span class="o">.</span><span class="na">class</span><span class="o">);</span> <span class="no">LOGGER</span><span class="o">.</span><span class="na">info</span><span class="o">(</span><span class="s">"Received record from Topic-Partition '{}-{}' with Offset '{}' -&gt; Key: '{}' - Value '{}'"</span><span class="o">,</span> <span class="n">incomingKafkaRecord</span><span class="o">.</span><span class="na">getTopic</span><span class="o">(),</span> <span class="n">incomingKafkaRecord</span><span class="o">.</span><span class="na">getPartition</span><span class="o">(),</span> <span class="n">incomingKafkaRecord</span><span class="o">.</span><span class="na">getOffset</span><span class="o">(),</span> <span class="n">incomingKafkaRecord</span><span class="o">.</span><span class="na">getKey</span><span class="o">(),</span> <span class="n">incomingKafkaRecord</span><span class="o">.</span><span class="na">getPayload</span><span class="o">());</span> <span class="c1">// Commit message</span> <span class="k">return</span> <span class="n">message</span><span class="o">.</span><span class="na">ack</span><span class="o">();</span> <span class="o">}</span> </code></pre> </div> <p>The schema is retrieved by the deserializer using the global ID written into the message being consumed from the Service Registry. <strong>Done!</strong></p> <h2> ๐Ÿ“‘ Summary </h2> <p>Your Quarkus applications integrate easily with the Service Registry and Apache Kafka to build your event-driven architecture. Thanks all these components you could get the following main benefits:</p> <ul> <li>Ensure consistent schema use between your client applications.</li> <li>Help to prevent data errors at runtime.</li> <li>Define a governance model in your data schemas (versions, rules, validations).</li> <li>Easy integration with client applications and components.</li> </ul> <p>You could invest and analyse more to adapt or build your event-drive architecture with these components in the following reference links.</p> <ul> <li><a href="https://www.apicur.io/registry/docs/apicurio-registry/1.3.3.Final/index.html" rel="noopener noreferrer">Getting started with Service Registry</a></li> <li><a href="https://developers.redhat.com/blog/2020/06/11/first-look-at-the-new-apicurio-registry-ui-and-operator/" rel="noopener noreferrer">First look at the new Apicurio Registry UI and Operator</a></li> <li><a href="https://quarkus.io/blog/kafka-avro/" rel="noopener noreferrer">How to Use Kafka, Schema Registry and Avro with Quarkus</a></li> <li><a href="https://quarkus.io/blog/avro-native/" rel="noopener noreferrer">Using Avro in a native executable</a></li> </ul> <h2> ๐Ÿ”ฌ Show me the code </h2> <p>Everything seems great and cool, but you want to see how it is really working โ€ฆ then this <a href="https://github.com/rmarting/kafka-clients-quarkus-sample" rel="noopener noreferrer">GitHub repository</a> is your reference.</p> <p>Enjoying API eventing ๐Ÿ˜ƒ!!!</p> <h2> ๐ŸŽ‰ Bonus Track </h2> <p>Quarkus 1.10.5.Final includes the native compilation of the Avro schemas capability. This command will compile as native my sample application:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight shell"><code>โฏ mvn package <span class="nt">-Dquarkus</span>.native.container-build<span class="o">=</span><span class="nb">true</span> <span class="nt">-Dquarkus</span>.native.container-runtime<span class="o">=</span>podman <span class="nt">-Pnative</span> </code></pre> </div> <p>But this is other story to write in other post! ๐Ÿ˜Ž</p> howto quarkus apicurio strimzi Connecting Apicurio Registry with secured Strimzi clusters Jose Roman Martin Gil Tue, 08 Dec 2020 09:15:00 +0000 https://dev.to/rmarting/connecting-apicurio-registry-with-secured-strimzi-clusters-24ne https://dev.to/rmarting/connecting-apicurio-registry-with-secured-strimzi-clusters-24ne <p><a href="https://www.apicur.io/registry/">Apicurio Registry</a> is a datastore for sharing standard event schemas and API designs across API and event-driven architectures. Apicurio Registry decouples the structure of your data from your client applications, and enables you to share and manage your data types and API descriptions at runtime. Decoupling your data structure from your client applications reduces costs by decreasing overall message size, creates efficiencies by increasing consistent re-use of schemas and API designs across your organization.</p> <p>Some of the most common uses cases where Apicurio Registry helps us are:</p> <ul> <li>Client applications can dynamically push or pull the latest schema updates to or from Apicurio Registry at runtime without needing to redeploy.</li> <li>Developer teams can query the registry for existing schemas required for services already deployed in production.</li> <li>Developer teams can register new schemas required for new services in development or rolling to production.</li> <li>Store schemas used to serialize and deserialize messages, which can then be referenced from your client applications to ensure that the messages that they send and receive are compatible with those schemas.</li> </ul> <p><a href="https://www.apicur.io/">Apicurio</a> provides a open sourced Schema Registry, ready to be involved in this scenario.</p> <p>Apicurio Registry includes a set of pluggable storage options to store the APIs, rules and validations. The <a href="https://kafka.apache.org/">Kafka-based</a> storage option, provided by <a href="https://strimzi.io/">Strimzi</a>, is suitable for production environments when persistent storage is configured for Kafka clusters running on <a href="https://www.openshift.com/">OpenShift</a>. </p> <p>In production environments security is not an option and it is something that must be provided by Strimzi for the different components connected to. Security will be defined by:</p> <ul> <li> <strong>Authentication</strong> to ensure a secure client connection to the Kafka cluster.</li> <li> <strong>Authorization</strong> to define which users have access to which resources.</li> </ul> <p>This blog post describes the high level topics to keep in mind to connect Apicurio Registry to a secure Apache Kafka cluster.</p> <h2> ๐Ÿค– OpenShift Operators </h2> <p>Strimzi and Apicurio Registry provides a set of OpenShift Operators, available through <a href="https://operatorhub.io/">Operator Hub</a>, to manage the life cycle of each component. Operators are a method of packaging, deploying, and managing OpenShift applications. </p> <p>Strimzi Operators provide a set of Custom Resources Definitions (CRD) to describe the different components of the Kafka deployment (Zookeeper, Brokers, Users, Connect, โ€ฆ) These objects will provide the API to manage our Kafka cluster.</p> <p>Strimzi Operators manage the authentication, authorization and users life cycle with the following custom resources:</p> <ul> <li> <a href="https://strimzi.io/docs/operators/latest/using.html#type-Kafka-reference">Kafka Schema</a>: Declares the Kafka topology and features to use. This object is managed by the <a href="https://strimzi.io/docs/operators/latest/using.html#using-the-cluster-operator-str">Cluster Operator</a>.</li> <li> <a href="https://strimzi.io/docs/operators/latest/using.html#type-KafkaUser-reference">KafkaUser Schema</a>: Declares a user for an instance of Strimzi, including the authentication, authorization and quotas definitions. This object is managed by the <a href="https://strimzi.io/docs/operators/latest/using.html#assembly-using-the-user-operator-str">User Operator</a>.</li> </ul> <p><a href="https://www.apicur.io/registry/docs/apicurio-registry/1.3.3.Final/getting-started/assembly-installing-registry-openshift.html#installing-registry-operatorhub">Apicurio Registry Operator</a>provides a set of CRD to describe the different components of the Apicurio Registry deployment (storage, security, replicas, โ€ฆ) These objects will provide the API to manage our Apicurio Registry instance.</p> <p>Apicurio Registry Operator manages the Apicurio Registry life cycle with the following custom resources:</p> <ul> <li> <a href="https://github.com/Apicurio/apicurio-registry-operator/blob/master/deploy/crds/apicur.io_apicurioregistries_crd.yaml">ApicurioRegistry Schema</a>: Declares the Apicurio Registry topology and main features to use. This object is managed by the Apicurio Operator.</li> </ul> <h2> ๐Ÿ‘ฎ Authentication </h2> <p>Strimzi supports the following authentication mechanisms:</p> <ul> <li>SASL SCRAM-SHA-512</li> <li>TLS client authentication</li> <li>OAuth 2.0 token based authentication</li> </ul> <p>These mechanisms are declared in the <code>authentication</code> block in the <code>Kafka</code> definition for each listener. Each listener will implement the authentication mechanism defined so the client applications must authenticate with the mechanism identified.</p> <p>How to activate each mechanism in the Strimzi cluster is described below.</p> <p>On the other hand we need to identify in the Apicurio Registry the authentication mechanism activated in the Strimzi cluster. Apicurio Registry only allows the following authentication mechanisms:</p> <ul> <li>SCRAM-SHA-512</li> <li>TLS</li> </ul> <h3> Strimzi Authentication </h3> <p><strong>Using SCRAM-SHA-512</strong></p> <p>The following <code>Kafka</code> definition declares a Kafka cluster secured with <code>SCRAM-SHA-512</code> authentication for the secured listener (TLS):<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>apiVersion: kafka.strimzi.io/v1beta1 kind: Kafka metadata: name: my-kafka spec: kafka: listeners: tls: authentication: type: scram-sha-512 </code></pre> </div> <p>Applying this configuration will create a set of ใŠ™๏ธ secrets where TLS certificates are stored. The secret we need to know to allow the secured connections is declared as <code>my-kafka-cluster-ca-cert</code>. We will need this value later.</p> <p>The following <code>KafkaUser</code> definition declares a Kafka user with <code>SCRAM-SHA-512</code> authentication:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>apiVersion: kafka.strimzi.io/v1beta1 kind: KafkaUser metadata: name: service-registry-scram labels: strimzi.io/cluster: my-kafka spec: authentication: type: scram-sha-512 </code></pre> </div> <p>Applying this configuration a new ใŠ™๏ธ secret is created, with the same name of the user, where credentials are stored. This secret contains the generated password to authenticate to the Kafka cluster.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ oc get secrets NAME TYPE DATA AGE service-registry-scram Opaque 1 4s </code></pre> </div> <p><strong>Using TLS</strong></p> <p>The following <code>Kafka</code> definition declares a Kafka cluster secured with TLS authentication for the secured listener (TLS):<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>apiVersion: kafka.strimzi.io/v1beta1 kind: Kafka metadata: name: my-kafka spec: kafka: listeners: tls: authentication: type: tls </code></pre> </div> <p>Applying this configuration will create a set of ใŠ™๏ธ secrets where TLS certificates are stored. The secret we need to know to allow the secured connections is declared as <code>my-kafka-cluster-ca-cert</code>. We will need this value later.</p> <p>The following <code>KafkaUser</code> definition declares a Kafka user with TLS authentication:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>apiVersion: kafka.strimzi.io/v1beta1 kind: KafkaUser metadata: name: service-registry-tls labels: strimzi.io/cluster: my-kafka spec: authentication: type: tls </code></pre> </div> <p>Applying this configuration will create a ใŠ™๏ธ secret, with the same name of the user, where user credentials are stored. This secret contains the valid client certificates to authenticate to the Kafka cluster.<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ oc get secrets NAME TYPE DATA AGE Service-registry-tls Opaque 1 4s </code></pre> </div> <h3> Apicurio Registry Authentication </h3> <p>Identifying the authentication mechanism activated in the Strimzi cluster, we need to deploy the Apicurio Registry defining accordingly the <code>ApicurioRegistry</code> definition.</p> <p><strong>NOTE</strong> : Apicurio Registry can only connect, at the time of writing this blog, to Strimzi_tls_ listener (normally in 9093 port), whatever the authentication mechanism is activated in that listener. It means that the <code>boostrapServers</code> property in <code>ApicurioRegistry</code> must point to that listener port:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>apiVersion: apicur.io/v1alpha1 kind: ApicurioRegistry metadata: name: service-registry spec: configuration: persistence: "streams" streams: bootstrapServers: "my-kafka-kafka-bootstrap:9093" </code></pre> </div> <p><strong>Using SCRAM-SHA-512</strong></p> <p>The following <code>ApicurioRegistry</code> definition declares a secured connection with a user with <code>SCRAM-SHA-512</code> authentication:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>apiVersion: apicur.io/v1alpha1 kind: ApicurioRegistry metadata: name: service-registry spec: configuration: persistence: "streams" streams: bootstrapServers: "my-kafka-kafka-bootstrap:9093" security: scram: user: service-registry-scram passwordSecretName: service-registry-scram truststoreSecretName: my-kafka-cluster-ca-cert </code></pre> </div> <p>The values that we need to identify in this object are:</p> <ul> <li> <strong>user</strong> : Name of the user to connect.</li> <li> <strong>passwordSecretName</strong> : Name of the secret where the password is saved.</li> <li> <strong>truststoreSecretName</strong> : Name of secret with the CA certs of the Kafka cluster deployed.</li> </ul> <p><strong>Using TLS</strong></p> <p>The following <code>ApicurioRegistry</code> definition declares a secured connection with a user with TLS authentication:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>apiVersion: apicur.io/v1alpha1 kind: ApicurioRegistry metadata: name: service-registry spec: configuration: persistence: "streams" streams: bootstrapServers: "my-kafka-kafka-bootstrap:9093" security: tls: keystoreSecretName: service-registry-tls truststoreSecretName: my-kafka-cluster-ca-cert </code></pre> </div> <p>The values that we need to identify in this object are:</p> <ul> <li> <strong>keystoreSecretName</strong> : Name of the user secret with the client certificates.</li> <li> <strong>truststoreSecretName</strong> : Name of secret with the CA certs of the Kafka cluster deployed.</li> </ul> <h2> ๐Ÿ›‚ Authorization </h2> <p>Strimzi supports authorization using <code>SimpleACLAuthorizer</code> globally for all listeners used for client connections. This mechanism uses Access Control Lists (ACLs) to define which users have access to which resources.</p> <p>Denying is the default ACL if authorization is applied in the Kafka cluster. That requires to declare the different rules for each user that wants to operate with the Kafka cluster.</p> <p>The following <code>Kafka</code> definition activates authorization in the Kafka cluster:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>apiVersion: kafka.strimzi.io/v1beta1 kind: Kafka metadata: name: my-kafka spec: kafka: authorization: type: simple </code></pre> </div> <p>ACLs are declared for each user in the <code>KafkaUser</code> definition in the <code>acls</code> section. That section includes a list of resources to declare, each of them as a new rule:</p> <ul> <li> <strong>Resource Type</strong> : Identifies the type of object managed in Kafka, objects such as topics, consumer groups, cluster, transaction ids and delegation tokens.</li> <li> <strong>Name of the resource</strong> : Identifies the resource where to apply the rule. It could be defined as literal, to identify one resource, or a prefix pattern, to identify a list of resources.</li> <li> <strong>Operation</strong> : Which kind of operation is allowed to do. A full list of operations available for each resource type is available <a href="https://strimzi.io/docs/operators/latest/using.html#acl_rules">here</a>.</li> </ul> <p>To allow the Apicurio Registry user to work successfully with our secured Strimzi cluster we must declare the following list of rules to specify what the user is allowed to do:</p> <ul> <li>Read its own consumer group.</li> <li>Create, read, write and describe on global ids topic (<em>global-id-topic</em>).</li> <li>Create, read, write and describe on storage topic (<em>storage-topic</em>).</li> <li>Create, read, write and describe on its own local changelog topics.</li> <li>Describe and write transactional ids on its own local group.</li> <li>Read on consumer offset topic (<em>__consumer_offsets</em>).</li> <li>Read on transaction state topics (<em>__transaction_state</em>).</li> <li>Idempotently write on cluster.</li> </ul> <p>ACLs will be similar to the following definition:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code> acls: # Group Id to consume information for the different topics used by the Service Registry. # Name equals to metadata.name property in ApicurioRegistry object - resource: type: group name: service-registry operation: Read # Rules for the Global global-id-topic - resource: type: topic name: global-id-topic operation: Read - resource: type: topic name: global-id-topic operation: Describe - resource: type: topic name: global-id-topic operation: Write - resource: type: topic name: global-id-topic operation: Create # Rules for the Global storage-topic - resource: type: topic name: storage-topic operation: Read - resource: type: topic name: storage-topic operation: Describe - resource: type: topic name: storage-topic operation: Write - resource: type: topic name: storage-topic operation: Create # Rules for the local topics created by our Service Registry instance # Prefix value equals to metadata.name property in ApicurioRegistry object - resource: type: topic name: service-registry- patternType: prefix operation: Read - resource: type: topic name: service-registry- patternType: prefix operation: Describe - resource: type: topic name: service-registry- patternType: prefix operation: Write - resource: type: topic name: service-registry- patternType: prefix operation: Create # Rules for the local transactionalsIds created by our Service Registry instance # Prefix equals to metadata.name property in ApicurioRegistry object - resource: type: transactionalId name: service-registry- patternType: prefix operation: Describe - resource: type: transactionalId name: service-registry- patternType: prefix operation: Write # Rules for internal Apache Kafka topics - resource: type: topic name: __consumer_offsets operation: Read - resource: type: topic name: __transaction_state operation: Read # Rules for Cluster objects - resource: type: cluster operation: IdempotentWrite </code></pre> </div> <p>Activating authorization in Strimzi has no effect in the <code>ApicurioRegistry</code> definition, because this is only related to the correct ACL definitions in <code>KafkaUser</code> objects.</p> <h2> ๐Ÿ“‘ Summary </h2> <p>Apicurio Registry includes the security capabilities to connect to secure Strimzi clusters, so your production environment could complain about your security requirements easily. This blog post demonstrates that these technical components in your event-driven architecture are concerned with security requirements and allow you to apply successfully.</p> <p>To deeper understand and analysis, please, refer to the following references:</p> <ul> <li><a href="https://strimzi.io/docs/operators/latest/using.html#assembly-using-the-user-operator-str">Using the User Operator</a></li> <li><a href="https://strimzi.io/docs/operators/latest/using.html#assembly-securing-kafka-brokers-str">Security options for Kafka</a></li> <li><a href="https://www.apicur.io/registry/docs/apicurio-registry/1.3.3.Final/getting-started/assembly-intro-to-the-registry.html">Introduction to Apicurio Registry</a></li> </ul> <p>Enjoying API eventing ๐Ÿ˜ƒ!!!</p> howto apicurio strimzi operators containers - The GNOME Shell Extension to manage podman containers Jose Roman Martin Gil Sun, 25 Oct 2020 08:00:00 +0000 https://dev.to/rmarting/containers-the-gnome-shell-extension-to-manage-podman-containers-3dp6 https://dev.to/rmarting/containers-the-gnome-shell-extension-to-manage-podman-containers-3dp6 <p>In my daily I usually start linux containers with <a href="https://podman.io/">podman</a> to have easily and quickly tools; such as databases, brokers or systems. This method allow me to avoid to install locally and administrate them locally.</p> <p>To manage these linux containers I have usually local scripts with the arguments, parameters and set up to my use cases (I forget very easily the commands โ€ฆ yes, I know! the <code>history</code> command could help me but I am lazy ๐Ÿ˜). These scripts include the typical options to start, stop, delete and so on. For many of my colleagues use these kind of commands are very common, however for me it is a little tedious and bored ๐Ÿ˜’ so to have a graphical tool will be great and better for me ๐Ÿ˜‹.</p> <p>So here is when I found a great tool to integrate with my Fedora laptop โ€ฆ</p> <h2> containers - the GNOME shell extension to the rescue ๐Ÿš‘ </h2> <p><a href="https://extensions.gnome.org/extension/1500/containers/">Containers</a> is a gnome-shell extension to manage linux containers, run by <a href="https://podman.io/">podman</a>. A simple menu allows us to execute the most typical actions ๐Ÿ“‹, such as:</p> <ul> <li>start</li> <li>stop</li> <li>remove</li> <li>pause</li> <li>restart</li> <li>top resources: opens the <code>top</code> command output ๐Ÿ“ˆ (user,cpu,elapsed,time,command) in a new terminal.</li> <li>shell: opens a shell in a new terminal ๐Ÿ’ป.</li> <li>stats: open statistics ๐Ÿ“ˆ (cpu,memory,networking,io) in a new terminal with updating live.</li> <li>logs: following logs in a new terminal ๐Ÿ“„.</li> </ul> <p>The menu also showed most of the inspect info โ„น๏ธ of the container, such as:</p> <ul> <li>status: running, stopped, exited, โ€ฆ</li> <li>id</li> <li>image</li> <li>command</li> <li>Created time</li> <li>Started time</li> <li>IP address</li> <li>ports</li> </ul> <p>A sample screenshot ๐Ÿ“ท of this amazing tool is similar to:</p> <p><a href="http://blog.jromanmartin.io/imagescontainers-gnome-shell-extension/containers-pod-view.png"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--h_cljE9---/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://blog.jromanmartin.io/images/containers-gnome-shell-extension/containers-pod-view.png" alt="" title="container view"></a></p> <h2> Managing containers </h2> <p>The extension manages the current pods created in your local environment, basically from <code>podman ps -a</code> command. So the first time to add containers you must to start them with the right arguments and setup for your use case.</p> <p>For example to start a local MongoDB instance, the command could be similar to:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ podman run -d -p 27017:27017 --name mongodb mongo 649cc435939a66537e11686c8d400c83250de5314b6735a8ade2a00a0a49b8b2 </code></pre> </div> <p>Or to start a local MariaDB instance could be similar to:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ podman run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=mypass --name mariadb mariadb:latest 744cfc9b4013f4f0db111aa96dd1ae4cf53bbd85f920c19470bef857b0836846 </code></pre> </div> <p><strong>๐Ÿ‘‰ Note:</strong> The <code>-d</code> argument starts detached the pod from the terminal (similar to execute in background).</p> <p>These commands will start two new pods as we could check with:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 744cfc9b4013 docker.io/library/mariadb:latest mysqld 28 seconds ago Up 28 seconds ago 0.0.0.0:3306-&gt;3306/tcp mariadb 649cc435939a docker.io/library/mongo:latest mongod 40 minutes ago Up 40 minutes ago 0.0.0.0:27017-&gt;27017/tcp mongodb </code></pre> </div> <p>These pods will be showed in the shell-menu as:</p> <p><a href="http://blog.jromanmartin.io/imagescontainers-gnome-shell-extension/containers-list.png"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--Ik7Guvd2--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://blog.jromanmartin.io/images/containers-gnome-shell-extension/containers-list.png" alt="" title="containers list"></a></p> <h2> Managing pods </h2> <p><a href="https://github.com/containers/podman-compose">podman-compose</a> allows to start pods (a group of containers as an unit), very useful when you have to compose a set of containers in one place.</p> <p>For example, the following <code>docker-compose-kafka.yml</code> file describes a pod definition to start an Apache Kafka topology instance (zookeeper + broker):<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>version: '3' services: zookeeper: image: strimzi/kafka:0.20.0-kafka-2.5.0 command: [ "sh", "-c", "bin/zookeeper-server-start.sh config/zookeeper.properties" ] ports: - "2181:2181" environment: LOG_DIR: /tmp/logs kafka: image: strimzi/kafka:0.20.0-kafka-2.5.0 command: [ "sh", "-c", "bin/kafka-server-start.sh config/server.properties --override listeners=PLAINTEXT://0.0.0.0:9092 --override advertised.listeners=PLAINTEXT://localhost:9092 --override zookeeper.connect=zookeeper:2181" ] depends_on: - zookeeper ports: - "9092:9092" environment: KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092 KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 LOG_DIR: /tmp/logs </code></pre> </div> <p>To start up this pod, we could use the following command:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ podman-compose -f docker-compose-kafka.yml -t 1podfw -p kafka up -d podman pod create --name=kafka --share net -p 2181:2181 -p 9092:9092 e21fb80e3e5ce6f156253277b14fbb66b013afdcae9fec468c42c6afde3a6668 0 podman run --name=kafka_zookeeper_1 -d --pod=kafka --label io.podman.compose.config-hash=123 --label io.podman.compose.project=kafka --label io.podman.compose.version=0.0.1 --label com.docker.compose.container-number=1 --label com.docker.compose.service=zookeeper -e LOG_DIR=/tmp/logs --add-host zookeeper:127.0.0.1 --add-host kafka_zookeeper_1:127.0.0.1 --add-host kafka:127.0.0.1 --add-host kafka_kafka_1:127.0.0.1 strimzi/kafka:0.20.0-kafka-2.5.0 sh -c bin/zookeeper-server-start.sh config/zookeeper.properties c6eba2decd63be4f28366546c96e00842d84442b48fdcc97a691a058cccd46dd 0 podman run --name=kafka_kafka_1 -d --pod=kafka --label io.podman.compose.config-hash=123 --label io.podman.compose.project=kafka --label io.podman.compose.version=0.0.1 --label com.docker.compose.container-number=1 --label com.docker.compose.service=kafka -e KAFKA_LISTENERS=PLAINTEXT://0.0.0.0:9092 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 -e LOG_DIR=/tmp/logs --add-host zookeeper:127.0.0.1 --add-host kafka_zookeeper_1:127.0.0.1 --add-host kafka:127.0.0.1 --add-host kafka_kafka_1:127.0.0.1 strimzi/kafka:0.20.0-kafka-2.5.0 sh -c bin/kafka-server-start.sh config/server.properties --override listeners=PLAINTEXT://0.0.0.0:9092 --override advertised.listeners=PLAINTEXT://localhost:9092 --override zookeeper.connect=zookeeper:2181 87a8e215f0ccf48a5e976444b5d4e7650879a4ab9e4a4c0ce5a88138e7eb99fe 0 </code></pre> </div> <p>Now you have an Apache Kafka pod instance up and running ๐Ÿ’ช.</p> <p><a href="http://blog.jromanmartin.io/imagescontainers-gnome-shell-extension/containers-pod-kafka-list.png"><img src="https://res.cloudinary.com/practicaldev/image/fetch/s--bX74UkQI--/c_limit%2Cf_auto%2Cfl_progressive%2Cq_auto%2Cw_880/http://blog.jromanmartin.io/images/containers-gnome-shell-extension/containers-pod-kafka-list.png" alt="" title="Pod Apache Kafka list"></a></p> <h2> Summary </h2> <p>This amazing GNOME shell extension will help you to manage easily your local linux containers, and since I started to use it โ€ฆ I feel more productive ๐Ÿ˜†.</p> <p>Enjoy it!</p> <h2> References </h2> <ul> <li><a href="https://extensions.gnome.org/extension/1500/containers/">Containers</a></li> <li><a href="https://github.com/rgolangh/gnome-shell-extension-containers">gnome-shell-extension-containers GitHub repo</a></li> <li><a href="https://podman.io">podman</a></li> </ul> howto fedora32 podman tools Blue / Green deployments in OpenShift with Eclipse JKube Jose Roman Martin Gil Fri, 23 Oct 2020 08:00:00 +0000 https://dev.to/rmarting/blue-green-deployments-in-openshift-with-eclipse-jkube-1i82 https://dev.to/rmarting/blue-green-deployments-in-openshift-with-eclipse-jkube-1i82 <p><a href="https://martinfowler.com/bliki/BlueGreenDeployment.html">Blue Green Deployment</a> is an application release model very well-known that gradually transfers user traffic from a previous version of an application or microservice to a nearly identical new releaseโ€”both of which are running in production.</p> <p>The old version of the application or microservice is identified with a color (e.g: Blue) while the new version is identified with the other color (e.g: Green). This model allows to manage the production traffic from one color (old version) to the other (new version), and the old version can standby in case of rollback or pulled from production and updated to become the template upon which the next update is made ๐Ÿ’ซ.</p> <p>This model requires a Continuous Deployment model or pipeline ๐Ÿ“จ to orchestrate the promotion between each color and manage the down times or rolling process. CD pipelines could be implemented using the capabilities provided by Eclipse JKube ๐ŸŽ‡๐Ÿ’ก.</p> <p><a href="https://www.eclipse.org/jkube/">Eclipse JKube</a> is a collection of plugins that are used for building container images using Docker, JIB or S2I build strategies. In addition, Eclipse JKube generates and deploys Kubernetes/OpenShift manifests (YAML configuration files) at compile time too.</p> <p>This article describes an approach to integrate a Blue/Greendeployment with Eclipse JKube. This approach is based in<a href="https://maven.apache.org/guides/introduction/introduction-to-profiles.html">Maven Profiles</a>and filtering resources. Of course, this is only an integration sample and it is open to have other approaches. For example, I would like to dig into the <a href="https://www.eclipse.org/jkube/docs/kubernetes-maven-plugin#profiles">Eclipse JKube Profiles</a> as other approach ๐Ÿ‘. However, feel free to add your ideas, comments or suggestions ๐Ÿ’˜.</p> <h2> Identifying Blue / Green Version </h2> <p>Every time we want to deploy a new version it is needed to identify the right color to be used to deploy it. This process should be done in the CD pipeline before to deploy our next application version.</p> <p>For example we could use a variable called <code>app.bg.version</code> in the <code>pom.xml</code> file. To declare each version a Maven Profile could help us. This variable will be used to filter some resources at deployment time.</p> <p>Sample Maven profile for Blue Deployment:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>&lt;profile&gt; &lt;id&gt;blue&lt;/id&gt; &lt;properties&gt; &lt;app.bg.version&gt;blue&lt;/app.bg.version&gt; &lt;/properties&gt; &lt;/profile&gt; </code></pre> </div> <p>Sample Maven profile for Green Deployment:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>&lt;profile&gt; &lt;id&gt;green&lt;/id&gt; &lt;properties&gt; &lt;app.bg.version&gt;green&lt;/app.bg.version&gt; &lt;/properties&gt; &lt;/profile&gt; </code></pre> </div> <p>This value could also be used to define the version of the image built. A sample of that could be using the <code>jkube.generator.name</code> property in the <code>pom.xml</code> file as:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>&lt;jkube.generator.name&gt;${project.artifactId}-${app.bg.version}:${project.version}&lt;/jkube.generator.name&gt; </code></pre> </div> <h2> Defining Blue/Green Objects </h2> <p>Each Blue and Green version should use its own resources at deployment time to avoid replace or redeploy the other deployment. To do that we would use<a href="https://www.eclipse.org/jkube/docs/kubernetes-maven-plugin#_resource_fragments">Eclipse JKube resource fragments</a>to declare specific definitions of some Kubernetes or OpenShift objects.</p> <p>In our case we will declare a custom deployment object creating a <code>deployment.yml</code> file in the <code>jkube</code> folder. This custom deployment will override some properties to use the active version at deployment time.</p> <p>A sample of this file could be similar to:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>metadata: name: ${project.artifactId}-${app.bg.version} labels: group: ${project.groupId} project: ${project.artifactId} version: ${project.version}-${app.bg.version} provider: jkube spec: template: spec: containers: - env: - name: APP_BG_VERSION value: ${app.bg.version} </code></pre> </div> <p>To access the version deployed a service is needed to map it. This service should be aligned with the right version. A <code>service.yml</code> file in the <code>jkube</code> folder similar to the next one could be similar to:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>metadata: name: ${project.artifactId}-${app.bg.version} labels: group: ${project.groupId} project: ${project.artifactId} version: ${project.version}-${app.bg.version} provider: jkube expose: "true" spec: selector: deploymentconfig: ${project.artifactId}-${app.bg.version} </code></pre> </div> <h2> Deploying the Blue Version </h2> <p>Now, we could deploy the Blue version easily using the Maven Profile. For example, using the <a href="https://www.eclipse.org/jkube/docs/openshift-maven-plugin">OpenShift Maven Plugin</a> the command will be similar to:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ mvn clean package oc:resource oc:build oc:apply -Pblue </code></pre> </div> <h2> Deploying the Green Version </h2> <p>On the other hand, we could deploy the Green version easily using the Maven Profile. For example, using the <a href="https://www.eclipse.org/jkube/docs/openshift-maven-plugin">OpenShift Maven Plugin</a>the command will be similar to:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ mvn clean package oc:resource oc:build oc:apply -Pgreen </code></pre> </div> <h2> Blue/Green Resources Deployed </h2> <p>This process generates the following objects:</p> <ul> <li>Image Streams ๐Ÿ“ท for each version: </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ oc get is NAME IMAGE REPOSITORY TAGS UPDATED kafka-clients-quarkus-sample-blue default-route-openshift-image-registry.apps-crc.testing/amq-streams-demo/kafka-clients-quarkus-sample-blue 1.0.0-SNAPSHOT,1.2.0-SNAPSHOT 38 hours ago kafka-clients-quarkus-sample-green default-route-openshift-image-registry.apps-crc.testing/amq-streams-demo/kafka-clients-quarkus-sample-green 1.1.0-SNAPSHOT,1.3.0-SNAPSHOT 38 hours ago </code></pre> </div> <ul> <li>Build Config ๐Ÿ‘ท for each version: </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ oc get bc NAME TYPE FROM LATEST kafka-clients-quarkus-sample-blue-s2i Source Binary 4 kafka-clients-quarkus-sample-green-s2i Source Binary 2 </code></pre> </div> <ul> <li>Deployments โœจ for each version: </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ oc get dc NAME REVISION DESIRED CURRENT TRIGGERED BY kafka-clients-quarkus-sample-blue 7 1 1 config,image(kafka-clients-quarkus-sample-blue:1.0.0-SNAPSHOT) kafka-clients-quarkus-sample-green 5 1 1 config,image(kafka-clients-quarkus-sample-green:1.1.0-SNAPSHOT) </code></pre> </div> <ul> <li>Services ๐Ÿ‘€ for each version: </li> </ul> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ oc get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kafka-clients-quarkus-sample-blue ClusterIP 172.25.180.88 &lt;none&gt; 8181/TCP 38h kafka-clients-quarkus-sample-green ClusterIP 172.25.87.170 &lt;none&gt; 8181/TCP 38h </code></pre> </div> <p>Thanks of all these objects we could manage both versions at the same time, and in combination with a CD pipeline and other resources could manage the traffic to activate the right version (Blue or Green) to the final users.</p> <p>For example, we could have an OpenShift route to balance the application between each versions. That route could be similar to:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>kind: Route apiVersion: route.openshift.io/v1 metadata: name: kafka-clients-quarkus-sample-lb labels: app: kafka-clients-quarkus-sample spec: to: kind: Service name: kafka-clients-quarkus-sample-blue weight: 100 </code></pre> </div> <h2> Rolling from Blue to Green </h2> <p>To roll from <strong>Blue</strong> to <strong>Green</strong> version it is only to path the load balancer route (in the case of OpenShift) as:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ oc patch route kafka-clients-quarkus-sample-lb --type=merge -p '{"spec": {"to": {"name": "kafka-clients-quarkus-sample-green"}}}' </code></pre> </div> <h2> Rolling from Green to Blue </h2> <p>To roll from <strong>Green</strong> to <strong>Blue</strong> version it is only to path the load balancer route (in the case of OpenShift) as:<br> </p> <div class="highlight js-code-highlight"> <pre class="highlight plaintext"><code>โฏ oc patch route kafka-clients-quarkus-sample-lb --type=merge -p '{"spec": {"to": {"name": "kafka-clients-quarkus-sample-blue"}}}' </code></pre> </div> <h2> Show me the code </h2> <p>If you want to test and verify this approach, I developed a sample case in one of my favorite<a href="https://github.com/rmarting/kafka-clients-quarkus-sample/tree/feature/b-g-deployment-strategy">GitHub repo</a>. This repo includes amazing frameworks as <a href="https://quarkus.io/">Quarkus</a>, <a href="https://avro.apache.org/">Schemas Avro</a>and <a href="https://kafka.apache.org/">Apache Kafka</a> in a small <a href="https://en.wikipedia.org/wiki/Event-driven_architecture">Event-Driven Architecture</a>.</p> <p>Enjoy it! ๐Ÿ’ช</p> <h2> References </h2> <ul> <li><a href="https://www.eclipse.org/jkube/">Eclipse JKube</a></li> <li><a href="https://github.com/eclipse/jkube">Eclipse JKube GitHub</a></li> <li><a href="https://www.redhat.com/en/topics/devops/what-is-blue-green-deployment">What is Blue Green deployment?</a></li> <li><a href="https://en.wikipedia.org/wiki/Blue-green_deployment">Wikipedia - Blue-Green deployment</a></li> </ul> howto openshift kubernetes jkube New Minishift Cheat Sheet! Jose Roman Martin Gil Mon, 05 Oct 2020 07:00:00 +0000 https://dev.to/rmarting/new-minishift-cheat-sheet-5555 https://dev.to/rmarting/new-minishift-cheat-sheet-5555 <p>Is OpenShift 3 โ€œlegacyโ€ ๐Ÿ‘ต ๐Ÿ‘ด? Maybe, however in some organizations is already the current <a href="https://en.wikipedia.org/wiki/Platform_as_a_service">Platform as a Service</a> deployed and running in production.</p> <p>From time to time I have to work on that platform, reviewing deployments, architectures, or designing migration approaches to OpenShift 4 using operators. Whatever it is the cause, I need to run locally an OpenShift 3 cluster, and here<a href="https://docs.okd.io/3.11/minishift/getting-started/index.html">minishift</a> helps me a lot โ€ฆ so:</p> <p>๐ŸŽ‰ I am pleasure to announce that a new Cheat Sheet is available ๐ŸŽ‰</p> <ul> <li><a href="https://dev.to/cheat-sheets/minishift">Minishift Cheat Sheet</a></li> </ul> <p>I hope this new cheat sheet helps you when you need to install, deploy and operate your own local OpenShift 3 cluster.</p> <p>My full list of Cheat Sheets are available for your records <a href="https://dev.to/cheat-sheets">here</a>. As usual, comments, ideas, PRs are welcomed!</p> <p>Happy reading !!! ๐Ÿ“š</p> howto cheatsheet minishift tutorial