DEV Community: frameworkz

Data Lakes

frameworkz — Tue, 27 Apr 2021 11:39:44 +0000

Our digital universe doubles in size annually - and is predicted to exceed 44 trillion gigabytes by the end of 2021. Up to 90% of this data is semi-structured or unstructured, presenting challenges in storing, maintaining, and processing such a vast volume. Here is where a data lake comes in.

What is a Data Lake?

A data lake is a central storage repository that holds a vast amount of data in a raw formula from many different sources. It can store data that is unstructured, semi-structured, or structured. Data can be kept in a flexible format for future use as a data lake will identify the data via metadata tags to enable fast retrieval when needed.

There isn’t a limit on storage capacity as clusters of data can exist on-premises or in the cloud.

Why use Data Lakes

Data lakes have become an essential part of many big data initiatives because they offer more flexible and easier options to scale when working with significant volumes of data, especially if it is being generated at a high velocity - such as app activity data. Web, sensor, and app activity data are increasingly prevalent, and so interest in data leaks is also growing at speed.

To determine if your company needs a data lake, let’s explore the following indicators:

How Structured is your Data?

If you’re processing a large volume of semi-structured or unstructured data, it can be extremely draining on time resources to do this without a data lake. Storing mass volumes of data that isn’t structured will require extensive data preparation; this is especially true for event-based data such as clickstream or server logs.

Is Data Retention a Problem?

Storing large volumes of data in a database can be an expensive task. This can lead to much fiddling about with data retention - either limiting the period in which historical data is held or trimming certain fields of the data to control costs. If your business struggles to strike the right balance between retaining data for analytical purposes versus deleting data to control costs, then a data lake solution.

Is your Use Case Experimental or Predictable?

What you intend to do with the data is really the determining factor in whether a data lake is the best solution. Suppose you want to build reports or dashboards that will essentially be created through running a predetermined set of queries against tables that are updated regularly. In that case, you may be better off looking at a data warehouse.

However, for experimental use cases - like predictive analytics and machine learning, it can be challenging to know in advance what data you will need, which is where a data lake could be more beneficial.

Benefits and Risks of using Data Lakes

As with all business solutions, there are benefits and risks when using data lakes.

Benefits in Using a Data Lake:

Aids with advanced analytics and product ionizing
Cost-effective scalability and flexibility
Great value from unlimited data types
Significantly reduced long-term cost of ownership
Permits economic file storage
Quickly adaptable
Centralisation of different content sources
Flexible access to the data worldwide - due to the cloud system

Risks of Using Data Lake:

May lose momentum and relevance over time
The prominent risk involved while designing data lake
Increases storage and computes costs
Because there is no account of the lineage of findings by previous analysts, there is no way to get insights from others who have worked with the data
Access control and security can be at risk
Unstructured data could lead to unusable data, disparate and complex tools, and enterprise-wide collaboration.

Modular Spring Boot Development

frameworkz — Mon, 15 Mar 2021 09:15:39 +0000

On a basic level, Spring Boot provides a fast way to build applications by looking at your classpath and at the beans you have configured, makes reasonable assumptions about what you are missing, and adds those items. With Spring Boot, you can focus more on business features and less on infrastructure.

I’ll cover some of the main concepts of Spring before we dig into the Modular aspects a little below.

Dependency Injection (DI) (Inversion of Control) – To give the application more flexibility, control is given to an external controller through events instead of the application taking control sequentially.

Beans & ApplicationContext – In Spring these are the two kinds of objects containers. Beans are managed and configured by a BeanFactory. This BeanFactory is a root interface for accessing the Spring container that configures and manages the beans. Implementations use lazy loading, which means that Beans are only instantiating when directly called.

ApplicationContext is the central interface within a Spring application for providing configuration information to the application. It's actually a superset of BeanFactory used for more complex applications that need event propagation, declarative mechanisms, and integration with aspect-oriented features of Spring.

Modularizing a Spring Boot Application

Every software project comes to a point where the code should be broken up into modules. This is because modular systems architecture allows for shorter development times, along with improved code management, and the deployment of different solutions for different requirements. These modules may reside within a single code base or could be modules that each reside within their own particular code base.

Modules in Spring Boot

In relation to Spring Boot, a module is a set of Spring components loaded into the application context.

A module can be a business module that provides some business services for the application, or it can be a technical module that provides cross-cutting concerns for several other modules or the entire application. Additionally, these modules could be part of a monolithic codebase or split up into multiple build modules using Maven or Gradle.

Creating Modules - Some Options

The base for a Spring Module is a @Configuration-annotated class along the lines of Spring’s Java configuration feature. There are several ways to define what beans should be loaded by such a configuration class.

@ComponentScan
The simplest method to create a module is by using the @ComponentScan statement on a configuration class:

@ComponentScan
public class BlogPostsFilter {
 {

As the configuration class is picked up by an importing mechanism, it will search in all classes in the package and load an instance of each class that is annotated with one of Spring’s annotations into the application context.

@bean Definitions
Beans are objects that form the backbone of your application, and they are defined, assembled, and managed by the Spring IoC container. These beans are created by the configuration metadata supplied to the container.

public JavaConfigBean javaConfigBean(){
       return new JavaConfigBean();
   }

This above configuration class is imported creating a TransferService instance that will be inserted into the application context.

The method of creating a module demonstrates what beans are actually loaded by having a single location to look at. The flipside of using @ComponentScan, which can be scattered through annotations of all classes in the package.

@ConditionalOn... Annotations
To achieve more control over which components should be loaded into the application context, then @ConditionalOn... annotations is the answer. This is how Spring loads the exact beans required.

@ConditionalOnClass(SpringTemplateRank.class)

Importing Modules - Some Options

The next step in using modular Spring Boot development is once the module is created then we will need to import it into the application. There are several methods of doing this:

@import...Annotations
will import the class and all beans that come with it. Any new bean added to the package will be automatically found in our context. And we still have explicit control over the configurations we are using.

@Import(JavaConfig.class)

@Enable... Annotations
Spring Boot already comes with a set of annotations that help developers to configure applications. The @Enable annotation allows for features to be enabled and configured. Some examples:

@EnableScheduling
@EnableAsync
@EnableWebSocket
@EnableJpaRepositories
@EnableTransactionManagement

Auto-Configuration
Spring also comes with an auto-configuration feature. Although super useful there are issues that may arise too. For example, auto-configuration is especially useful when building a cross-cutting solution that will be used across several Spring Boot applications. However, the limitations because all beans registered due to auto-configuration will be available to all contexts, which might not be what we want.

To enable a module for auto configuration, put the file META-INF/spring.factories into the classpath:
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
com.xyz.libx.autoconfigure.LibXAutoConfiguration

Configuring a Module

@ConfigurationProperties works best with hierarchical properties giving Spring Boot quality support for binding external configuration parameters to a Spring bean. The Spring framework uses standard Java bean setters, so we must declare setters for each of the properties.

Modular development at run time through plugins

For deployment flexibility, run time modularization provides additional benefits over strictly modular development.

With plugins, development and deployment can be separated so new plugins can be developed and deployed without changing the main application.

Thus different teams can work on different aspects of a system and expand it later without changing the Spring Boot application, it provides significant time-saving in quality assurance (QA) and deployment. Different customers can use a different set of plugins matching their special requirements.
This is possible with on-premise deployment as well as on the cloud.

The main difference between modular Spring development and Plugins based development is related to the packaging of the application.

With modular Spring development, the entire application is packaged together, libraries that use the same third-party components may cause “dependency hell” that must be resolved at the packaging time, ultimately only a specific version of the third-party library may be selected. All logical components are known at this point in time, this means that implementations must be also provided at this point.

With plugins-based development the host application and each of the plugins is separately packaged, a properly developed plugin system allows overcoming dependency hell as each plugin is separately loaded.

Logical components are unknown at this point, this allows deciding on the relevant implementation at the deployment time. Additional features, perhaps from a completely different application domain can be added through plugins.

A properly designed and implemented plugins-system should allow any part of a system to be implemented through plugins such as API interfaces, business logic, and persistence.

Injectable plugins using FlexiCore Boot and Spring Boot

FlexiCore Boot is an open-source library available on GitHub and documented at support.wizzdi.com and Wizzdi. FlexiCore artifacts are available on Maven Central.

Support for inter-injectable plugins can be added to any existing Spring Boot application or used on a newly created application. FlexiCore encourages and supports building solutions as sets of interconnected plug-ins, each developed using Spring Boot APIs, services, and paradigms.

A plug-in may depend on other plug-ins, these plugins are injected (@AutoWire) into dependent plugins. All the components of a system may be defined as plug-ins.

In FlexiCore Plug-ins have no limitations to what they define or provide, including the interfaces to front-end devices (APIs), the domain model (Database structure), business services, and Spring event bus.

Unlike Spring modules, the main application has no ‘knowledge’ of what plugins will be available to it at run-time. Plugins are deployed by placing them in a predefined location on the server.

Thus, the above mentioned requirement for supporting different deployments for different customers is addressed at deploy time and not at build time. Simply dispatch a different set of plugins to create a different ‘flavor’ of a system.

Moreover, different versions of the same plugin can be concurrently deployed. The exact version may be selected by the caller, for instance, the client mobile or browser application accessing the API a plugin exposes.

Examples

Adding plugins support
@EnableFlexiCorePlugins
This should be added near the @SpringBootApplication annotation. Once added, the application will load and use plugins stored in a configurable folder on the server.

Adding REST API support (in plugins)
@EnableFlexiCoreRESTPlugins
Once added (on the same location), plugins can now create REST API endpoints. Clients (mobile, browsers, and other servers).

Adding support for JPA in plugins
@EnableFlexiCoreJPAPlugins
Although not strictly considered as plugins, FlexiCore allows model definition jars built to conform to JPA standard to be placed in a predefined entities folder. The domain model defined in these ‘plugins’ is available to plugins and also to other entities definition plugins that may be dependent on them.

Adding support for health messages in plugins
@EnableFlexiCoreHealthPlugins
Allows plugins use the standard annotations Spring provides for actuators and health.

Summary

Spring’s great value is multifaceted - firstly as an integration framework. Then from having stable and familiar libraries to implement all layers of applications. And with all the benefits as a dependency injection container and MVC framework.

Modular architectures encourage cohesive program units, which should be easier for developers to read and understand. This Modularisation of a codebase enables parallelisation in its build process, making it easy for modules to develop into stand-alone services if their boundaries are clear.

On large projects i think modularisation could be essential to offering structure to the code base while also reducing crossover and duplication of developers resources.

With FlexiCore plugins, Spring Modularity can be implemented at run time with requiring new builds of the complete monolithic application.

Why should you do coding competitions

frameworkz — Mon, 22 Feb 2021 11:47:42 +0000

When I was younger I used to compete in coding competitions but never really made the grade. I’ve got a lot of respect for these guys who get to the top through dedication and perseverance, there are some cool prizes to be won and massive kudos to be earned.

Even though i was competing i was never a serious player but i loved the sense of community, competitivity aside i was just so in awe that so many different people were involved in something i was interested in, and good at.

The more I thought about competing over the years of these competitions the more I realised how useful they were for my own development and career. So here’s my attempt to cover why they are a good idea, followed by a list of competitions.

Enhance problem-solving skills
Brush up programming skills
Develop team player skills
Encourage to write cleaner and better code
Good brain exercise
Employment opportunities
Join you with community
Prizes and money

CodeChef

CodeChef hosts a programming contest at the start of each month and two smaller programming challenges at the middle and end of the month. There’s a practice section with problems that can be solved using different languages, a good way to get started.

Leetcode

Leetcode runs weekly and biweekly contents as well as daily challenges and a Bug Bounty Program with many eligible for rewards.

Topcoder

Topcoder provides challenges as well as competitive programming competitions as well as paying community members for their work on projects. They also sell community services to business clients when required.

Coderbyte

A challenge library with a simple entry, you don’t even need to register to start testing your skills here. You can just jump in and start solving coding challenges using the online editor. You can really test your knowledge and skill by difficulty or from specific companies like Google and Facebook. Lots of starter courses for newbies and budding developers to learn new skills. Codebyte is also a hunting ground for employers so the Interview Kits is a great addition.

HackerRank

HackerRank has some great challenges from Competitive Challenges, Hackathons, Hiring Challenges (Jobs) and University Challenges to test your skills. You can also host your own hackathon.

Company Code Contests:
Microsoft Imagine Cup
Google
Facebook

More competition websites:
Project Euler
Exercism
Code Wars
Sphere Online Judge
CodinGame
Code Wars Central

Let me now if i missed something.

Should developers do their own testing?

frameworkz — Fri, 19 Feb 2021 13:36:46 +0000

Let's start with stating that the majority of developers don’t fully test their own code, usually because they are backed up by QA testers. It's easier to allow QA testers to use their time to find defects rather than spending the time testing it themselves. Maybe they are too busy or the company prefers this method and to have their dev team churning out tasks. There’s also an arrogant aspect at play potentially, as some developers believe their work is without flaws but this could (and has) lead to serious problems that affect business.

With many dev teams testing their own work and not using QA testers, this topic inspires a heated debate from both sides. I’m going to cover both sides, and hopefully help you understand the strengths and weaknesses from both approaches.

Let's start with developer-based testing:
Most teams (I believe) do require its developers to run unit testing (at least) and many also require automated, integrated, code-based tests. The problem is that this does take time away from working on new features and code and we as developers are attracted to the next shiny thing to work on. Testing could be built-in if it covers the entire codebase but again this takes time away from new tasks.

In some ways these developers are correct in saying that testing can waste time, even if the testing process is automated. Testing can be seen to slow down the development and productivity.

Another reason testing is not done is purely because some developers don’t have an understanding of how the entire application works. Or they don’t believe mistakes exist in their code, arrogant I know but we’ve all met people like this, in some cases they are actual colleagues.

On the other side there are many developers who rigorously test their code using coded tests. A sense of pride in their work means they want to fully test their own work to ensure its defect free, or at least to ensure that any QA testers don’t find faults or defects in their code. Pride, is should be noted is seen as one of the sins of developers :)

Personally, I don't enjoy testing because i find it boring and repetitive. And i know many who think the same but we need to ensure that developer testing can be a productive part of your QA.

QA-based testing
On the QA side I have a lot of respect for QA tests because it takes balls to question a developer's work and deal with any defensiveness that can come back. There are some very good QA testers out there doing a great job but the best ones need to be confident when reporting bugs to the dev team.

They also tend to have much more knowledge of the application in its entirety with all its nuances, including regression history. I think they also help to ensure the developers are honest and hard working, because in some ways they are covering the work on a dev manager testing and providing oversight of the entire development process.

As their job is based on finding faults, QA testers push an application harder than a developer. A simple test and success will not be enough as they seek ways creatively to execute tests that may not occur to the developer. Proof is everything and bugs are not fixed unless proven, something many developers don’t go far enough when testing their own code.

Developers and QA testers should, of course, work in conjunction with each other. Code testing works well but a QA tester can run more human tests that are unexpected and ensure the application is more robust.

Another aspect of QA is the creative side, most of us see testing as boring repetitive work but from dealing with QA testers i’ve realised that creativity is an important part of the QA process. Based on their knowledge they can think of many different ways to test an application and compare to previous testing. Documentation created from testing is super useful for training and even in some cases for client usage.

So which is better?
The answer is neither, a combination of both approaches works best. With many Dev teams constantly and quickly releasing code there is a need for QA to play its part. I understand that speed to market is a crucial factor but stability is also key to success.

Another aspect i see is companies outsourcing their QA rather than having it run inhouse and together with the dev team. Several new businesses such as Global and Digivante are some of the companies where outsourcing QA or crowd testing is growing. I do understand the hole they fill in the space but is it a good move for companies to outsource QA testing?