DEV Community: Ana Vasiliuk

The State of Open Source: A FOSS Backstage 2022 Event Report

Ana Vasiliuk — Thu, 07 Apr 2022 14:44:31 +0000

The post is written in collaboration with @floord and originally published on aiven.io/blog

FOSS Backstage was back in Berlin last month, this time as a hybrid event with the first day being fully online. Floor and I ventured out (virtually) to hear some talks, and then wrote this blog post to share some of the interesting developments in the open source community and the key themes of the event: open source sustainability, contributor experience, open source licensing, and diversity and inclusion.

Open Source Sustainability

How to keep an open source project going once the novelty wears off was definitely one of the hottest topics at this year’s event. The schedule was packed with sessions on sustainability, ranging from broader discussions about funding, project health, and more practical content like tips and best practices.

Patricia Leu and Marie Gutbub of Prototype Fund, a funding program for open source software at the Open Knowledge Foundation Germany, kicked off the second day with the keynote about ways of keeping projects running besides funding with some inspiring examples.

Dr. Wolfgang Gehring, FOSS Ambassador at Mercedes-Benz AG and its IT-subsidiary Daimler TSS, shared how they use GitHub Sponsors to support the projects they rely upon. Wolfgang’s story of finding the right person to sign off on the program (ultimately it was their CISO) is certainly an interesting one for folks advocating for similar efforts at their organizations.

Regina Nkemchor Adejo of IKEA raised the importance of mentorship as a way to sustain a project, help recruit newcomers, and retain skilled contributors.

Jennifer Fernick, Anne Bertucio and Christopher Robinson (aka CRob) talked about CVEs, and “the dance” with researchers reporting vulnerabilities. The gist of it: all complex software has flaws. What matters most is how you respond and patch. OSS vulnerabilities take years to detect - a typical vulnerability on GitHub goes undetected for over 4 years - but take mere days to exploit. The 2021 Open Source Security and Risk Analysis Report found that 84% of FOSS codebases had at least one vulnerability, with the average having 158 per codebase.

It’ll come as no surprise to anyone that the Log4J vulnerability was often referenced at FOSS backstage. As was the famous XKCD comic, when illustrating the risk of adding software to your stack that is maintained by an individual, in their free time.

Licensing

Relicensing is an issue that is looming in nowadays open source communities, with recent cases of companies deciding to go no-OSS or OS contributors sabotaging (their) projects. How do we as contributors or users prepare for such changes?

In his session, Dotan Horovits of Logz.io talked about three recent case studies: Elasticsearch, Grafana, and colors.js and faker.js - two popular NPM libraries by the same author - and shared some advice for dealing with such situations:

Companies shouldn’t think of OSS as a business model. If you’re a vendor choosing the OSS path, build a sustainable business model that is independent of the core of the OS project.
If you’re a maintainer, don’t expect material compensation or establish a vendor entity around the open source project if you want to get compensated.
If you’re a user, manage your third party licensing exposure, choose projects with less restrictive licenses, and continue to monitor your dependencies for changes that have implications for your tech stack.

Ms. Danese Cooper in her talk “Frenemies in FOSS: How to deal with Charter Conflicts and competing efforts” brought some excellent points around forking a project, and how to deal with a so-called hostile fork.

Contributor Experience

Contributors are the stars of an open source project. Here at Aiven, we’re very strong advocates for supporting open source contributors, their mental health, accessibility needs, etc. We were happy to hear a lot of great ideas and conversations around this topic. One of the sessions by Ruth Cheesly (Acquia) even called for making this the year of the contributor experience!

Ways to support contributors

Firstly, think of the fundamentals: safety & inclusion, supporting the contributors in a positive way, and dealing with inappropriate behavior quickly and decisively.

Secondly, source contributions don’t just include code: pull requests, answering questions on forums and chats, written content, docs, events, onboarding - these are all examples of contributions, and OS communities should strive to recognise them as such.

Finally, think of how to “fix the leaks” in your OS community by clearly documenting ways for people to contribute, onboarding new members, highlighting specific open tasks in newsletters/message boards, and regularly reaching out to contributors for feedback.

If you need more practical advice for keeping your contributors happy, Jim Hall of Hallmentum shared how they’ve engaged with the FreeDOS community, which has been running the project for almost three(!) decades. Spoiler: if you want your project to grow, make your community members feel like they’re part of it. That means you need to communicate to coordinate, make it easy for new people to contribute, and make your community inclusive to non-code contributions as well.

The impact of accessibility

Making your contributor experience a smooth one also depends on how accessible your project is, including its documentation. Writing accessible documentation is no easy task, and luckily, Alexandra White from Google shared some of the best practices:

Make sure your docs are written in inclusive language with the vocabulary that avoids stereotyping and is free from descriptors that are discriminatory or otherwise excluding.
Check for colloquialisms, slang, or mental health language.
Write clear and more specific instructions: the more details, the better. For example, don’t forget to include simple things like checking if they’re using the right browser, log in instructions, etc.
Reduce visual cues (colors, patterns, images, font styles, directional words).

With talks by Chief Strategic Solutions at the United Nations Office of Information and Telecommunication Technology Maurizio Gazzola, and Ingo Hinterding, Public Tech Lead at the CityLAB Berlin, it’s clear that FOSS becomes even more of a commodity in government and public administration as well.

Session recordings will be made available on the Plain Schwartz YouTube channel shortly. Keep an eye out for updates by following FOSS Backstage’s Twitter account.

Introduction to MySQL

Ana Vasiliuk — Wed, 23 Feb 2022 14:00:23 +0000

MySQL has been a key part of the open source database ecosystem for a long time, with many of the world’s largest and fastest-growing companies adopting it for their business-critical systems. If you’re looking for a way to get started with MySQL, this blog post has a good overview of its capabilities and common use cases, so read on!

What is MySQL?

MySQL is an open source relational database management system (DBMS). It is developed, supported, and distributed by Oracle Corporation and it has a vibrant community around it. Some of the key MySQL features include:

Portability: MySQL works on many different platforms and is configured using CMake
Works on top of various storage engines, with the option to add new engines relatively easily to provide a SQL interface to an in-house database
In-memory hash tables, which are used as temporary tables
Many data types, including JSON and spatial
Full operator and function support in the SELECT list and WHERE clause of queries
Full support for SQL GROUP BY and ORDER BY clauses and group functions
Flexibility and security
MySQL includes several clients and utilities written in many languages

What are the primary use cases for MySQL?

MySQL is the most widely adopted relational database management system and can be used for mission critical apps, dynamic websites, and as an embedded database for software, hardware, and appliances. Here are the examples of some of the most common use cases:

Online Transaction Processing (OLTP) – MySQL can be useful both in any application that requires a simple SQL engine and websites with heavy traffic. It adheres to ACID principles and its extensions to ANSI/ISO Standard SQL support a large volume of databases.
LAMP Stack – MySQL’s simple programming syntax makes it compatible with high-performance applications running on the LAMP stack (Linux, Apache, MySQL, and PHP/Python/Perl).
eCommerce – MySQL can process a high volume of requests in the shortest amount of time, so it’s good for managing customer data, catalogs, and transactions.
Fraud detection – Thanks to the high availability of MySQL, it’s able to analyze transactions in real time, along with trends or anomalous behavior, to prevent fraudulent activities.

These are just some examples of situations where you can use MySQL. As it’s a highly popular database management system and it’s open source, it has an extensive portfolio of extensions and community support. All this makes MySQL a flexible tool for many use cases.

How to get started with MySQL?

It’s relatively easy if you use Aiven for MySQL™. Whether you are building something new or looking for a modern platform for your existing applications, Aiven for MySQL™ is a friendly and scalable offering. Here’s how to get started:

Log in to the Aiven Console (or sign up for a 30-day trial if you haven’t done so yet!).
Create a new Aiven service by following these steps.
Click the service name in the list to go to the ”Overview” page. You can configure the service using our list of available parameters if needed.
Once the service is up and running, the light changes to green and the indicator says ” RUNNING “.

That’s it! You can explore the next steps on our Developer Portal - we have quite an extensive list of How-tos and resources available at https://developer.aiven.io/docs/products/mysql/.

Wrapping up

We hope you found this introduction useful and are ready to get started with MySQL. For more details about the tool, check out our resources below or the official MySQL docs.

Getting started with MySQL guide
Check out this blog post on the difference between MySQL and PostgreSQL
Find out about our open-source solution to MySQL backups and restoration MyHoard and check out the GitHub repository

Lastly, try out our fully-managed service, deployable in the cloud of your choice. Start with a 30-day trial and see if it works for your use case. As we mentioned, MySQL has a vibrant community to answer any questions you may have, or our team can help answer those questions for you too.

What is Uptime? Introducing the Developer Conference by Aiven

Ana Vasiliuk — Mon, 21 Feb 2022 12:43:36 +0000

You may have heard the news that we are hosting a developer conference this September, called Uptime. In this blog post, we’ll go over some of the frequently asked questions about the conference and its target audience. Before we jump into the details, a quick PSA: The Call for Presentations for Uptime is open, so if you already have a talk idea in mind, submit it by March 9 via the CFP form.

What is Uptime?

Uptime is the cloud data infrastructure conference with a focus on open source technologies. We intend to bring the open source data community together to share best practices and around building and operating complex systems at scale.

This is a developer conference and, as self-identified total geeks, we want to pack in as much technical content as possible. Of course, as it’s an in-person event, we’ll leave plenty of time for social interaction and stimulating conversations!

The event is happening on September 14-15 at Pakhuis de Zwijger in Amsterdam (the website is mostly in Dutch, sorry about that, unless you speak Dutch in which case you’re welcome). The venue is quite easily accessible by car and public transport, and it has hosted many technical events before us.

Who is Uptime for?

The conference is designed for developers, architects, data engineers, DevOps professionals, and anyone who is passionate about open source data. The focus is on use cases for open source tools like PostgreSQL®, Apache Kafka®, Redis™, Apache Flink®, and Grafana®, and best practices in the data engineering space and cloud infrastructure.

Here are some of the topics we plan to cover:

Cloud data architecture
Data engineering
Distributed systems
Developer tools (Observability, monitoring and alerting, SRE and performance)

Sounds interesting. How can I participate?

We have recently launched the Call for Presentations and are looking forward to receiving your compelling ideas! Our main priority is to provide the audience with the best possible learning experience, so the content is key to this event. We’re looking for your most exciting and informative talks on open source data tools, so if you have a proposal, submit it via the CFP form by March 9, 2022.

Even if you’re a first-time speaker or don’t have a specific idea in mind yet, don’t hesitate to take the leap. Don’t worry too much about whether we’ll accept your proposal - just submit it anyway and leave the rest to us. Our team is here to support you throughout the submission process, Reach out to us via uptime-speakers@aiven.io - we’d be happy to help you brainstorm ideas and offer guidance.

Wrapping up

We hope this sparks your interest in attending and/or speaking at Uptime this fall. If you need more information about the event, you can always reach us via uptime-hello@aiven.io. Hope to see you in September in Amsterdam!

Introduction to OpenSearch

Ana Vasiliuk — Thu, 27 Jan 2022 14:07:36 +0000

OpenSearch is a community-driven and truly open source search and analytics suite. Read on for a high-level look at its capabilities.

In this post, we’re going to provide a high-level overview of OpenSearch components and use cases that would be fitting for this open source search and analytics engine. Let’s dive right into it!

What is OpenSearch?

In short, OpenSearch is an open source alternative to Elasticsearch. It is a search and analytics suite that includes a search engine daemon, OpenSearch, NoSQL database, and a visualization interface. It offers a distributed, full-text search engine based on Apache Lucene with a RESTful API interface and support for JSON documents.

Let’s look at some of OpenSearch's components and concepts in more detail.

Clusters and nodes

OpenSearch is a distributed search and analytics engine, which means that you interact with its highly scalable clusters. Each cluster consists of one or more nodes that store your data and process search requests.

As mentioned above, the clusters are scalable. You can run a single node cluster, like your laptop, or multiple powerful machines. As your cluster grows, you can divide responsibilities across those machines. You can also tailor different servers to different types of tasks. Those machines that have fast disks could index and search data, while a server with a smaller disk, but high CPU, can be responsible for cluster’s state management.

If you want more details, you can check out the Cluster formation article on the Open Search project site.

Indices

Much like databases and tables in relational databases, OpenSearch uses indices to organize data. The data is distributed within your cluster by mapping each index to a primary shard, which is copied to one or more replica shards. This protects your data from hardware failure and provides additional capacity for read requests.

The Indices article on the Aiven Developer Portal provides a deeper dive into the topic of indices and shards.

REST API

For communication, OpenSearch relies on the REST API. You can use any programming language or client like cURL to send HTTPS requests.

Below is an example of adding a JSON document to an OpenSearch index by sending an HTTPS request, taken from the official OpenSearch project site.

PUT https://<host>:<port>/<index-name>/_doc/<document-id>
{
  "title": "The Wind Rises",
  "release_date": "2013-07-20"
}

To run a search for the document:
GET https://<host>:<port>/<index-name>/_search?q=wind

To delete the document:
DELETE https://<host>:<port>/<index-name>/_doc/<document-id>

Aggregations

One of the great things about OpenSearch is its powerful analytics engine. The engine can perform calculations, analyze your data, and extract statistics from it. This can come in handy when you try to analyze data in real-time and visualize it in OpenSearch Dashboards.

If you’re curious to learn more about OpenSearch aggregations, check out our article about Aggregations on the Developer Portal or the even more detailed one from the OpenSearch project site.

What are the primary use cases for OpenSearch?

With OpenSearch, you can easily ingest, search, aggregate, view, and analyze data. These capabilities are popular for use cases such as log analytics, real-time application monitoring, clickstream analytics, search backend, and more.

How to get started with OpenSearch?

Great news! You can get started with OpenSearch right in the Aiven Console and here’s how:

Choose the OpenSearch version, your cloud provider and location to deploy to, then choose which plan to use
Give your service a name
Click “Create Service”, and your shiny new OpenSearch database will start building.
While it does that, you can already visit the service overview page to see the details of the service.

To test your service and play around with your own data, you can start by connecting with cURL. If you’re hungry (pun intended) for test data, we’ve picked a sample recipe dataset that you can use to explore OpenSearch.

Wrapping up

If you’re looking for a powerful open source search and analytics engine, OpenSearch has pretty much everything you need to search and manage your data.

Check out our documentation for Aiven for OpenSearch on the Developer Portal
Get started with OpenSearch with our sample dataset
If you’re an Aiven for Elasticsearch user and are interested in moving to OpenSearch, check out Migrate Elasticsearch data to Aiven for OpenSearch for help in migrating your data
Find out how to get started with OpenSearch using NodeJS on the Aiven blog
Aiven for OpenSearch repositories on GitHub

And, finally, sign up for a free trial to start using Aiven for OpenSearch and follow us on Twitter to stay up-to-date with product and feature-related news.

Optimising Data Streaming Pipelines with Apache Flink and Apache Kafka: a panel recap

Ana Vasiliuk — Wed, 03 Nov 2021 09:36:24 +0000

Earlier this week we got to host a panel discussion at Flink Forward Global 2021, the conference on all things Apache Flink and stream processing. Being part of the conference for the first time (for me, as an attendee this time round 😉) was exhilarating and we really enjoyed interacting with the Flink community.

Our panellists, Olena Babenko (Aiven), Francesco Tisiot (Aiven), and Gyula Fora (Cloudera), sat down to discuss their favourite things about Flink SQL and how it’s a good fit in your data streaming pipeline with Kafka. We certainly got a lot of great questions from the audience and useful insights.

The panel recording is now available on the Flink Forward YouTube channel and we can’t wait for you to watch it! In the meantime, read on for a recap and some tips from our experts.

What makes Apache Kafka a great fit for streaming data and why does that fit together with Flink?

According to Francesco Tisiot, as we’re moving away from batch processing and going into streaming, we need a platform that is performant and scalable, and proven to be successful. Apache Kafka is the perfect match in this case with its beautiful features like Kafka Connect, allowing you to connect your Kafka instance to other systems. However, it often acts as a messenger, taking data from one place and pushing it to the other.

If you want to level up your game and perform analytics, Flink is a very powerful tool for that. It understands the architecture of Kafka (i.e. topics, partitions) and optimises the workload across those parameters.

Secondly, Flink supports a vast range of data platforms, both for input and output. This makes it a good choice for your data pipelines, no matter where your data sits. The combination of Flink and Kafka is truly powerful and takes your data streaming to the next level.

What are the common challenges of using both Flink & Kafka?

Working with Kafka and Flink services, we’re often faced with the issue of skewed data. For example, if you have 10 partitions in your Kafka service and only 1 partition has 5GB and the rest have 2MB. Usually this can happen because of a mismatch between the node keys. If there is a mismatch in requests, your Kafka and Flink service performance will suffer when they try to process the data.

According to Olena Babenko, you can avoid that by adjusting your metrics, such as maximum number of message bytes and patch sizes, to minimise the impact of the overload of data. Additionally, if you are the manager of the Kafka service, you can review your partitioning mechanism to ensure an even distribution of messages.

Where could Flink improve?

Our panellist had diverse opinions on this topic. As a Flink expert and long-term committer, Gyula Fóra mentioned that Flink has been greatly improving with each release. There is definitely no better time to join Flink than now, with a lot of new feature additions, great connectors, and cloud support. Kudos to the Flink community for working hard on adding new features at an incredible pace!

However, there are some things that could be improved on the SQL side of things. For example, good operational support for SQL queries could come in handy or a way of guaranteeing savepoint compatibility between SQL jobs. Additionally, as in Francesco’s case, telling a user exactly where errors in the SQL statements are could make it a better experience, especially for new Flink users.

The SQL client, although it has been improved a lot, is still a work in progress. Especially if we look at the SQL CLI, you want to make sure that, when you run real-world applications, logging and all other operations are configured correctly and set up for production. Most companies have a deployment stack around this and a regular Table API program is a better fit than the SQL CLI.

Another small issue with SQL applications is the difficulty to write a unit test on it, because it’s hard to make it isolated and fast. According to Olena, one tip would be to mock as many sources of your data as possible (e.g. data fakers) to try and perform in-memory testing. In the worst case scenario, you can also try using a file system or Dockerised tools as sources.

What are the alternatives to Flink?

According to our panelists, Flink is becoming more and more mainstream. In most cases, it has pretty much all you need for stream processing. In some cases, the developer experience in the company makes it impractical to get a new tool, e.g. when you already have a strong team with extensive Kafka experience. However, the downside of using platform-specific tools like kSQL is that you are forced to use Kafka as both source and sink. Flink, on the other hand, doesn’t have such a limitation and works over a broad tech ecosystem with its wide range of connectors.

If you’re new to Flink, what’s the most fun thing to try out?

Without a doubt, you should check out Flink documentation. According to our panellists, there is a lot of great content that will help you understand Flink’s capabilities. Secondly, we recommend trying the SQL client as a great way to get started with Flink. Thirdly, if you are looking for a streaming SQL interface, we just launched our Flink beta program and we can’t wait for you to try it out!

5 Reasons Why You Should Consider Presenting at Flink Forward Global Virtual 2020

Ana Vasiliuk — Wed, 24 Jun 2020 13:58:54 +0000

Written in collaboration with Flink Forward Event Manager, Laura Cuttiford.

At Flink Forward, we are grateful that the digital technology landscape has enabled us to embrace ‘the new now’, and present our conferences in a virtual capacity. Of course, this comes with its challenges, but overall we have been able to adapt to this new way of thinking, and in April 2020, we successfully hosted our first-ever virtual event to a global audience of around 1400 Apache Flink enthusiasts.

Following this success and with the health and safety of our community at the core of our activity, we present Flink Forward Global, October 19 - 22. A four-day conference, with two days of Flink training and two days of conference, including keynotes, breakout sessions, sponsor ‘areas’ and networking opportunities.

The Call for Presentations is currently open until June 28 (this Sunday), so we thought that we would offer some words of encouragement for those wishing to submit their stories alongside speakers from companies such as Netflix, Yelp, Pinterest, and AWS.

Here are 5 reasons why you should consider presenting at Flink Forward:

Opportunity to present your Flink experiences to an untapped and engaged audience
Removing the restriction of travel, allows people to attend Flink Forward, from anywhere in the world. We have experienced a significantly higher uptake of pre-registrations for this event and are excited for you to access a diverse audience of Flink enthusiasts from around the globe. In addition to this, we are working on a networking solution which will enable you to connect with attendees outside of your scheduled talk. Watch this space!

No cost for you or your employer
If you have previously been restricted with your public speaking posts, due to insufficient travel funding from your company, or struggled to dig deep enough in your own pockets for such events, this is no longer an issue! Speaking virtually means zero costs or wasted time due to travel. In fact, you can now boost your position as a core member of the Apache Flink community, by sharing your stories from the comfort of your own home.

Real-time audience feedback
A fully virtual audience allows us to monitor and track attendee activity much easier than before, which means you can see exactly how many people are attending your session as well as their feedback on your talk through the chat function and talk rating system.

Don’t feel comfortable giving a live talk? Pre-record it from the comfort of your home!
We offer a number of options for how you can contribute to the Flink Forward conference program. If you are not able to give a talk in person on the day, due to insufficient internet speed or other distractions, you can choose to pre-record your session and enjoy the conference and interact with fellow speakers and attendees without having to worry about your talk. We do however encourage you to conduct a live Q&A session after your talk ends.

Decreased carbon footprint
Flying and even day to day commuting creates a large imprint on your carbon footprint. If there is anything we have learnt in 2020, is that we can have the whole world at our fingertips, without the necessity of travel!

Of course, if you have any questions (or concerns!) before submitting your talk for Flink Forward, feel free to reach out to a member of our team hello@flink-forward.org.

Hope to see you virtually in October!

Resources
Submit your talk - CFP closes June 28!
Pre-register for Flink Forward Global
Follow us on Twitter
Follow us on Linkedin

Flink Forward Virtual Conference 2020 - what to expect?

Ana Vasiliuk — Thu, 27 Feb 2020 17:02:03 +0000

UPDATE: Flink Forward San Francisco 2020 is now a virtual event! Due to the ongoing COVID-19 outbreak, the physical event sadly had to be cancelled. However, we've spent the past couple of weeks sorting out the online event and I'm happy to share that it's now happening on April 22-24.

Are you interested in open source and data processing? If the answer is yes, you might want to check out the Flink Forward conference on April 22-24. It's an annual community event dedicated to stream processing and Apache Flink. This virtual event will gather over 500 software developers, engineers, data professionals, DevOps, etc. from AWS, Bird, Godaddy, Netflix, Lyft, Uber, Yelp and many more to share use cases, knowledge, and the latest updates in the stream processing space in a form of live webinars and Q&A sessions. Here's what you can look forward to:

Keynotes! Don't miss the chance to hear the inspiring sessions on the future vision of stream processing as well as how Cloudera, Splunk have been integrating Flink into their products & changing their data infrastructure.
"It’s finally here: Python on Flink & Flink on Zeppelin" by Jincheng Sun & Jeff Zhang, Alibaba. The Flink community has been doing a lot of great work around Flink's Python support - join to hear the latest development in PyFlink and what's coming in the future.
"Equality and inclusion as a way of developing diverse, resilient, and more relevant OSS" by Griselda Cuevas, Google & Markos Sfikas, Ververica. Equality and diversity are two very important topics in the open source community right now, and Flink is no exception. In this talk, Griselda and Markos discuss how can we grow a healthy and diverse community to maximise the value & adoption of an open source project.
"Flink SQL in 2020: Time to show off!" by Fabian Hueske & Timo Walther, Ververica. A few years ago, the Flink community started adding SQL support to ease and unify the processing of static and streaming data. In this talk, Fabian and Timo will demo the current streaming SQL possibilities as well as dive into the roadmap & upcoming features.
"Structured logging at Scale" by Hrishikesh Joshi & Mayank Verma, Lyft. As a highly reliable and distributed system, Lyft's ride-sharing platform requires the ability to troubleshoot applications and infrastructure performance in near real-time, which can be challenging. Hrishikesh and Mayank introduce how their team is tackling these challenges with a Flink-based near real-time logging platform.
"Real time identity graph" by Fakrudeen Ali Ahmed, Adobe. Hear how the identity team at Adobe uses Flink to build and update the identity graph in near real time with 50000+ events/second, 2B+ devices and serving 400 Million users. Fakrudeen will also share their journey of selecting Flink amongst other real-time data processing frameworks.

I've only covered a few beginner-level sessions in here and I suggest visiting the conference website for more exciting technical talks, use cases, interactive sessions for those who are interested in diving deeper.

Join the virtual conference here.