DEV Community: Eric See

Links to other Platforms

Eric See — Sun, 10 Dec 2023 14:26:38 +0000

https://medium.com/@ksericpro

How to collect IOT data, do magic and publish and sell enriched insights!!

Eric See — Fri, 10 Feb 2023 01:06:46 +0000

Stay tune. I will discuss the architecture and details on how to setup one.

How to supercharge your Microservices Architecture

Eric See — Fri, 10 Feb 2023 01:04:19 +0000

Stay tune..

Trottling
Websocket
Event Sourcing
SAGA
Per Database or Shared Database
Kafka + Redis as ESB
No-SQL or SQL for data store
Central Logging

How to prevent getting into trouble when using Open-Source plugins

Eric See — Fri, 10 Feb 2023 00:57:30 +0000

TWO Categories of Open Source

1. Permissive (CopyRight)

A permissive open source license is a non-copyleft open source license that guarantees the freedom to use, modify, and redistribute, while also permitting proprietary derivative works. Permissive open source licenses, lovingly referred to as “Anything Goes”, place minimal restrictions on how others can use open source components. That means that this type of license allows varying degrees of freedom to use, modify, and redistribute open source code, permitting its use in proprietary derivative works, and requiring nearly nothing in return in regards to obligations moving forward.

2. Restricted (CopyLeft)

When an author releases a program under a copyleft license, they make a claim on the copyright of the work and issue a statement that other people have the right to use, modify, and share the work as long as the reciprocity of the obligation is maintained. In short, if they are using a component with this kind of open source license, then they too must make their code open for use by others as well.

License Type(s)

1. MIT

The MIT license gives users express permission to reuse code for any purpose, sometimes even if code is part of proprietary software. As long as users include the original copy of the MIT license in their distribution, they can make any changes or modifications to the code to suit their own needs. MIT licenses allow commercial use. Users do not have to provide any other source code when releasing new software.

2. BSD-3/X11

The BSD 3-Clause License has two key requirements of those who want to use the licensed code. They’re actually the same as those of the MIT license. **_If you plan to copy, modify, or distribute any code licensed under BSD, you must include:

The full text of the license
The original copyright notice_**

Users of BSD-licensed code may:

Use the code commercially.
Modify the code.
Distribute reworked versions or copies of the code.
Place warranty

3. GNU GPLv2/v3

These GPL series are all copyleft licenses, which means that any derivative work must be distributed under the same or equivalent license terms. It is more restrictive than the Lesser General Public License and even further distinct from the more widely used permissive software licenses BSD, MIT, and Apache.

Any software created using an open-source component with a copyleft license must be released as open source as well. The result is that any software product containing an open source copyleft license, even when it’s only a few lines out of mountains of code, must make its entire source code available for free, along with the rights to modify and distribute it.

4. Apache 2.0

Apache License 2.0 can be easily used by other projects without any rewording in the license document itself.

The Apache License is also a permissive license. However, it has stringent terms when it comes to modifications. It requires you to explicitly list out all the modifications that you’ve done in the original software, i.e., you’re required to preserve modification notices.

You can use any Apache License 2.0 licensed software in your commercial products for free. However, you must not name your product in a way that it looks like an endorsement from Apache. You must also not use any of the Apache marks (like the multi-colored feather) anywhere in your product or its documentation.

Stay safe till we e-meet again!!

Developing Form Validations

Eric See — Thu, 09 Feb 2023 08:29:41 +0000

Validate process starts from form design

Very often we will need to create or update some forms before committing them to the backend.

Even before we create forms, keep open text fields to a mininum. Often, it is good to let the user select from predefine lists, radio choices, etc.

Validations both in backend and frontend

Validations of fields should be done on both frontend and backend. Alot of people implement validations on the frontend only. When someone uses postman or intercept the request and consume the APIs directly, it will create chaos in the database.

Common validations & code snippets

Some of the common validation checks are as follows:

Mandatory

//validate mandatory
Core_Utils.validateMandatory = function(value) {
  if (!value) {
    return true;
  } else {
    return false;
  }
}

Numeric

Core_Utils.isNumeric = function(value) {
    return !isNaN(value);
}

Email Format

//Validate Email
Core_Utils.validateEmail = function(email) {
    var re = /^(([^<>()\[\]\\.,;:\s@"]+(\.[^<>()\[\]\\.,;:\s@"]+)*)|(".+"))@((\[[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}])|(([a-zA-Z\-0-9]+\.)+[a-zA-Z]{2,}))$/;
    return re.test(email);
}

Minimum & Maximum Length Check

//Length Check
Core_Utils.validateLength = function(value, min, max) {
  if ((value>=min) && (value<max)) {
    return true;
  } else {
    return false;
  }
}

Whether field contains special characters

//Check whether content is safe
/*
< (open angle parenthesis)
> (close angle parenthesis)
• ' (single apostrophe)
• " (quotation mark)
• \' (backslash-escaped apostrophe)
• \" (backslash-escaped quotation mark)
• ( (opening parenthesis)
• ) (closing parenthesis)
• ; (semicolon)
• % (percentage)
*/
Core_Utils.isSanitized = function(content) {
    if (content!="") {
      var regex = /<|>|\'|\"|\(|\)|;|%/g;
      return !regex.test(content);
    }
    return true;
}

JSON content

//Check whether string is JSON
Core_Utils.isJson = function(str) {
    try {
        JSON.parse(str);
    } catch (e) {
        return false;
    }
    return true;
}

IP Address

//validate Ip address
Core_Utils.validateIPaddress = function(ipaddress) {
  if (/^(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$/.test(ipaddress)) {
    return true;
  }
  return false;
}

Version (major.minor.variant format)

//validate version
Core_Utils.validateVersion = function(version) {
  if (/^\d{1,2}\.\d{1,2}\.\d{1,2}$/.test(version)) {
    return true;
  } else {
    return false;
  }
}

Validation real-time or upon submit?

Personally, its more user friendly to display error message in real-time as the user types.

Happy Coding!

Average Monthly Salary in Singapore

Eric See — Mon, 25 Jul 2022 01:55:00 +0000

I thought of doing something different. Since I am from Singapore, this is a report by JobStreet stating the Average Monthly Salary. Take this as a pinch of salt!

What is Data Governance

Eric See — Mon, 25 Jul 2022 01:43:00 +0000

What is data governance?

There are 10 components under this umbrella:

Data Quality
Data Architecture
Data Modelling & Design
Data Storage & Operations
Data Security
Data Integration & Interoperability
Documents & Contents
Reference & Master Data
Data Warehousing & Business Intelligence
MetaData

A picture paints a thousand words:

Next, we need these key components to implement data governance.

People The data governance professionals, data stewards, and other key business and IT staff are the backbones of a data governance program. They establish and develop workflows to ensure that the enterprise data governance requirements are met.
Data strategy The data governance team plays a crucial role in the development and implementation roadmap of an organization’s enterprise data strategy. A data strategy is an executive document that provides high-level enterprise requirements for data and ensures that those requirements are met. Building an enterprise data strategy is a vital step in the organization’s data management journey.
Data processes Data governance programs need to establish key data processes for data management. These include data issue tracking or resolution, data quality monitoring, data sharing, data lineage tracking, impact analysis, data quality testing, and many others.
Data policies A data policy is a high-level set of one or more statements that state expectations and expected outcomes of data that influence and direct data habits at an enterprise level. Data governance programs establish data governance policies for data management. Policies include outbound data sharing, regulatory adherence, and many others.
Data standards & data rules A data standard provides a framework and an approach to ensure adherence to a data policy. An example of a data standard could be using the ISO 3166 standard for the definition of the codes for the names of countries, dependent territories, special areas of geographical interest, and their principal subdivisions. A data rule directs or constrains behavior to ensure adherence to data standards, which provides compliance wi****th data policies. An example of a data rule would be an organization that only allows country codes listed in the ISO 3166 Standard. Typically, organizations will look to establish data rules for master and reference data, data definitions and domain development, metadata management, classification, accessibility, and many others. A data governance program can leverage many data standards. Some of the more notable data standards include:
International Organization for Standardization (ISO): 3166, 19115, 11179
Dublin Core: A basic, domain-agnostic, most widely used metadata standard that can be easily understood and implemented.
Also Read: Top 10 Data Governance Tools for 2021
Data security Data security involves protecting digital data, such as those in a database, from destructive forces and unwanted actions of authorized and unauthorized users. These unwanted user activities refer to espionage, cyber-attack, or data breach.
Communications Data governance communications include all written, spoken, and electronic interactions with association audiences who need to know about the data governance team’s activities. A communication plan encompasses objectives, goals, and tools for all communications and should be part of a governance program from the very beginning. The plan identifies how to present governance and stewardship challenges and successes to the various stakeholders and the rest of the organization. The communications plan highlights the right business cases and presents their results.
Socialization Socialization of data governance is an important activity in any governance program. The data governance socialization plan is a plan that helps integrate data governance activities into an organization’s policies, internal culture, hierarchy, and processes. The plan is unique to the organization as it is tailored to its culture and standards of behavior.
Metrics/KPIs Establishing business metrics and key performance indicators (KPIs) for monitoring and measuring the overall business impact of the data governance program is vital to the program’s success. The metrics and KPIs must be measurable, tracked over time, and consistently measured the same way every year.
Technology The data governance program needs various technologies that make the process seamless and automated. Smaller data governance programs typically use the technology stack which they already have within their enterprise. Meanwhile, larger data governance purchase software that is specific to data governance and the functions it requires. It simplifies the process of capturing the required metadata, management of the metadata, automating the data stewardship workflows, decision trees, collaboration, and many other data governance functions.

Algorithm - Find Largest Upper Case Character when a same character in lower Case & Upper Case occurs in a string

Eric See — Sat, 16 Jul 2022 07:04:45 +0000

I encounter this in Codility, so going to share my solution.

findBiggestAlphabet = function(S){
    console.log(`Processing..`, S)
    const char_arr = [...S];
    var max_char = "";
    for(var i=0; i<S.length-1;i++){
        var char = S[i];
        var char_c = char.toUpperCase()
        //console.log(`Processing ${char}`);
        var new_arr = char_arr.slice(i+1, char_arr.length);

        var exist_cap = new_arr.find(x=>x===char_c)? true: false;
        if (exist_cap){
            if (char_c>max_char){
                max_char = char_c;
            }
        }
    }
    if (max_char) return max_char;
    return "NO";
}

var A_1 = "aabcDBA";
var A_2 = "dabcDBA";
var A_3 = "abcde";
console.log(findBiggestAlphabet(A_1));
console.log(findBiggestAlphabet(A_2));
console.log(findBiggestAlphabet(A_3));

Algorithm - How to derive Minimum Swap between 'a' or 'b' to break 3 consecutives same characters

Eric See — Sat, 16 Jul 2022 07:00:43 +0000

I encounter this in Codility, so going to share my solution.


findSwapCharacterCounts = function (S) {
    console.log(`Processing..`, S);
    var next = 0;
    while(next<S.length-1){
        var char = S[next]
        var next_char = S[next+1];
        if(next_char==char) {
           // console.log(`${char} same as ${next_char}`);
            if ((next_char == S[next+1]) && (next_char === S[next+2])){

                var end_char = S[next+2];
                var swap_char = end_char==="a"?"b":"a";
              //  console.log(`index=${next+2}, swap_char=${swap_char}`);
                var new_str = "";
                for (var j=0; j<S.length;j++){               
                    if (j==next+2){
                        new_str+=swap_char;
                    } else {
                        new_str+=S[j];
                    }
                }
                return findSwapCharacterCounts(new_str)+1;
            }

        } else {
           // console.log(`${char} different ${next_char}`);
        }
        next++;
    }
    return 0;
}

var B_1 = "baaaaa";
var B_2 = "baaaaaa";
var B_3 = "ababababa";
var B_4 = "abbbaaab";
console.log(findSwapCharacterCounts(B_1));
console.log(findSwapCharacterCounts(B_2));
console.log(findSwapCharacterCounts(B_3));
console.log(findSwapCharacterCounts(B_4));

Spark Use Cases

Eric See — Tue, 24 May 2022 01:19:58 +0000

https://www.qubole.com/blog/apache-spark-use-cases/

Web3 Modern Architecture

Eric See — Thu, 12 May 2022 03:56:13 +0000

Blockchain Layers

Eric See — Thu, 12 May 2022 01:02:29 +0000

There are 4 layers in blockchains:

https://medium.com/@nick.5montana/blockchain-layers-l0-l1-l2-l3-in-a-diagram-569162398db

Layer 0:
The ground floor. This is where the internet, hardware, and connections exist that allow layer 1s like Bitcoin to run smoothly. Layer 0s are allowing several things to happen:

1) Allow blockchains to interact with each other

A great example is Cosmos, which creates an ecosystem of interoperable blockchains thanks to its ‘Tendermint IBC’ (Inter-blockchain communication protocol).
For developers, this is massive. If a Dapp can function on one blockchain, it can function automatically on other blockchains so long as they’re built using the same layer 0. No need to invest more time and resources to build the same app on another chain.

2) Faster and cheaper transactions

With IBC, PoS consensus can be achieved across multiple chains resulting in finality times happening almost instantaneously (finality = when a block is approved, can’t be rolled back, and is considered irreversible). The result is faster and cheaper transactions on cross-chain exchanges.

3) Infrastructure for developers

Developers don’t need to start from scratch and build their blockchains from the ground up. Many features are pre-built and ready to be implemented immediately.

Layer 1:
Layer 1s are blockchains (Bitcoin and Ethereum) that process and finalize transactions on their own blockchain. This is where things like consensus (PoW, PoS) and all the technical details like block time and dispute resolution take place.

The most important three aspects of blockchains are conquering the blockchain trilemma: decentralization, security, and scalability. Still no single blockchain has nailed all three.

Layer 2:
Layer 2s are third-party integrations used in conjunction with layer ones to increase scalability and transactions per second (system throughput).

When you hear zero-knowledge rollups (zk rollups), side chains, or anything to do with speeding up transaction throughput, it’s likely layer 2.

Layer 3:
Layer 3 is the application layer. This is the UI that we as consumers actually interact with.