DEV Community: Pouya Eghbali

Kenshi Data Management

Pouya Eghbali — Mon, 02 Oct 2023 00:58:16 +0000

🕵️ What’s Big Data?

Have you ever wondered why Netflix effortlessly brings to your attention shows that you might probably like? How searching for a flight on Google magically populates your Web2 browsing experience with hotel deals in your intended country of destination? Don’t you love how your YouTube homepage is tailored to take you down deeper rabbit holes when it comes to the pursuit of your interests, be that coding, learning DeFi, or beating Elden Ring?
We take for granted the many day-to-day conveniences that these algorithms afford us, but there is one all-encompassing term that powers this invisible machine.

“Big Data” fragments information to make sense of the greater puzzle that is human behaviour. It helps businesses, data scientists, and researchers recognize enhanced patterns of user behaviour, translate actionable insights, improve fraud detection, and reduce costs associated by highly optimizing resource allocation.

In this article, we’ll discuss how prevalent the use of Big Data is in Web2, as well as how much of it can be applied to Web3 given the current limitations (and opportunities) of blockchain technology.

🔍 Recognize Patterns of User Behaviour

Businesses are inundated with an avalanche of data generated by user interactions across various platforms today. Big Data serves a solution in offering granular insights into customer data (i.e. click stream data, social media activity, and purchase history), such that businesses can tailor their user experience much better when navigating the overwhelming plethora of products and services offered.

🖱️ Click Stream Data

Businesses can gain insights into the preferences, which products are servers users are interested in for a given time period, understand how they navigate the landing page, and where they tend to drop off in the sales funnel.

Recognizing these patterns in data, can help businesses optimize and develop better product/service offerings for its customers to maximize the purchasing process, boost conversion rates, and better their user experience. Google’s Ads is a prime example of providing these actionable insights for businesses.

🥸 Granular Insights

Big Data analytics allows businesses to dissect and analyze user data at a granular level. It involves processing and scrutinizing every click, every interaction, and every transaction. This level of granularity is critical because it uncovers intricate details about user behavior that would otherwise remain hidden. Datadog offers a good solution in scaling these insights without compromising security.

💪 Competitive Advantage

In today's competitive landscape, businesses that harness the power of Big Data to recognize user behavior patterns gain a significant advantage. They can adapt quickly to changing consumer preferences, make data-driven decisions, and stay ahead of the competition. Adobe’s Analytics helps you track the customer journey, to prolong customer experiences within businesses.

📶 Network Analysis

Big data can examine network traffic patterns and identify anomalies that might suggest a cyberattack or unauthorized access to a network. This can help prevent data breaches and protect sensitive information. Allowing to quickly identify suspicious activities that may indicate a cybersecurity threat, such as malware infections, unauthorized access attempts, or unusual data transfers. Solarwinds services help many of the networks today using these modern techniques to preserve security.

🤖 Machine Learning Models

Big data leverages machine learning algorithms to continuously learn and adapt to new fraud tactics. These models can identify emerging fraud patterns, even those that are not explicitly programmed into the system, by analyzing historical data. Such patterns could include multiple large transactions within a short period, inputting the incorrect or transactions from unfamiliar locations. When these anomalies are detected, it triggers alerts for further investigation, potentially flagging fraudulent activities like credit card fraud or identity theft.

💳 Reduce Costs

🪵 Resource Allocation

By analyzing data on resource utilization, organizations can make more informed decisions about resource allocation. This is crucial in fields like healthcare, where big data can help hospitals optimize staff schedules, reducing labor costs while maintaining quality care.

⚡ Energy Consumption

In industries like utilities, big data can analyze energy usage patterns to identify opportunities for energy conservation. This can lead to significant cost savings through reduced energy consumption.

Marketing Efficiency

Through targeted marketing campaigns, big data reduces marketing costs by ensuring that advertisements reach the right audience. This minimizes spending on broad, less effective campaigns.

Blockchain data is not there yet

Despite the immense potential and promises that blockchain technology holds, it has not yet reached its full potential in addressing several critical issues related to blockchain data management and utilization. These challenges include:

👥 Sybil Of Services

One of the persistent challenges in the blockchain ecosystem is the "Sybil problem," where malicious actors can create multiple fake identities to manipulate the network. This issue can compromise the integrity and reliability of blockchain data, affecting various applications such as incentive structures, voting systems, reputation systems, and more.

💵 User Acquisition Costs

Many blockchain and DeFi applications struggle with the high cost of acquiring users. This is partly due to the complex and unfamiliar nature of blockchain technology, which can deter potential users from participating in blockchain-based services or applications.

🌎 Accessibility of Data

While blockchain technology promises transparency and immutability, accessing and extracting relevant data from blockchains can be challenging. Traditional methods of querying and analyzing data may not be suitable for blockchain systems, leading to inefficiencies and delays in data retrieval.

💶 Affordability for usability

The cost of interacting with blockchain networks, including gas fees and transaction costs, can be prohibitive, particularly for individuals and small-scale businesses. This high cost can hinder the widespread adoption of blockchain technology and limit its potential impact. We examine this much more closely with how Kenshi can build Power Apps on a Budget.

Addresses these Challenges with Kenshi

Kenshi, as a blockchain data indexing platform, aims to play a pivotal role in mitigating the aforementioned challenges and advancing the blockchain ecosystem. Here's how Kenshi aims to make a difference:

💳 Cost Efficiency

Kenshi offers a cost-efficient solution for indexing and accessing blockchain data. By optimizing data storage and retrieval processes, Kenshi helps reduce the overall cost of interacting with blockchain networks, making it more accessible to a wider audience.

📈 Convenience for DeFi Apps

Decentralized Finance (DeFi) applications rely heavily on real-time and historical blockchain data. Kenshi's indexing capabilities streamline data access for DeFi developers, making it easier to build powerful and efficient DeFi applications.

📶 IoT Accessibility

The integration of blockchain with the Internet of Things (IoT) is a promising avenue for tangible accessibility and data security. Kenshi's services facilitate efficient interaction between IoT devices and blockchain networks, ensuring that data from connected devices can be securely and seamlessly integrated into blockchain applications. As elaborated in the future vision for Kenshi.

🔢 Problem Solving

Kenshi acknowledges the riveting issues facing blockchain technology today and is dedicated to providing practical solutions. By addressing data-related challenges, Kenshi contributes to the overall maturation and widespread adoption of blockchain technology.

In summation, while blockchain technology holds immense promise, it still faces several hurdles in terms of data management, accessibility, and affordability. Kenshi emerges as a proactive player in the blockchain ecosystem, offering solutions to these challenges and supporting the evolution of blockchain technology into a more robust and accessible framework for a wide range of applications.

The Kenshi of Tomorrow: Web3 + IoT

Pouya Eghbali — Sat, 23 Sep 2023 00:56:26 +0000

Story of IoT

The emergence of Internet of Things (IoT) has unveiled a new realm for the technology with the desire to inspect, connect and interact with various devices.

Coined by Kevin Ashton, a British technology pioneer in 1999, it served as the foundation to describe systems where everyday objects could be uniquely identified, represented, and connected with the internet, enabling efficient streams of communication between ‘things’ and systems.

In the preliminary phase of the information age a paradigm shift was occurring in the Web 1.0 environment, when Ashton had presented the idea to executives at razor company Gillette, who were interested in partnering with Procter & Gamble as a solution to solve supply chain issues utilizing Radio Frequency Identification Tags (RFID), to help understand where items are scanned, placed, or sold. The phrase was met with… crickets, but Ashton remained well adept to keep sharing the ideas of uniting the internet with tangible objects for efficiency.

The phrase only became ubiquitous in 2008-2009 onwards, when the advancements of wireless communications, miniaturization of sensors, and the proliferation of internet connectivity paved the way for the concept of IoT.

Benefits of IoT

As IoT technology continued to evolve, its impact became increasingly significant. It revolutionized industries such as healthcare, agriculture, transportation, and logistics.

❤️‍🩹 Healthcare

In healthcare, IoT devices enabled remote patient monitoring, improved diagnostics, and enhanced the overall quality of care.

🧑‍🌾 Agriculture

In agriculture, IoT sensors helped irrigation, monitor crop health, and increase yields.

🛤️ Transportation and Logistics

Transportation and logistics benefited from IoT-enabled tracking and monitoring systems, leading to improved efficiency and reduced costs.

Kenshi of Today

In the rapidly evolving world of blockchain technology, Kenshi continues to assert its leadership as an avant-garde innovator. This foundation can now be completed with the new generation of Web 3.0 data consumption to simplify, manage, query, analyze, and automate the chasm between the web and the decentralized economy.

Reimagining the realms of opportunities for conceptualizing, deploying, and utilizing the suite of tools that capture a plethora of data from EVM-based blockchain networks. Developers can utilize the data infrastructure to support their blockchain ecosystems no matter if they are small or large in a cost-efficient manner.

The data infrastructure for Deep Index Sync, Deep Index GraphQL & MQL, and Deep Index R-API offer a more conventional experience to the current Web 2.0 landscape within minutes of creating an account.

As we had mentioned in our previous article, the data infrastructure landscape has evolved tremendously, the accessibility of on-chain data has been remarkable, yet costs continue to go higher and technical barriers exist which hinder data scientists, researchers, and engineers to illuminate their own actionable insights.** Kenshi’s pricing model has always remained transparent, predictable, and scalable**. Helping you to unlock revolutionary ways to access and utilize blockchain technology more effortlessly than ever!

Kenshi of Tomorrow

The integration of IoT with Kenshi’s data infrastructure allows boundless possibilities, customized to your preference and unique requirements to build applications that have a tangible use case impacting the real-world.

To reach critical mass, development of a toolkit is underway for building more interconnected systems with blockchain technologies. Kenshi would like to more broadly expand into being the IoT hub for developers to forge real-world applications and harness the affordable data stack Kenshi is building.

Supercharging data accessibility, integrations, and elevating in the realms to amplify blockchain data requests efficiently.

Need more concrete examples of what’s possible?

Look no further!

🛒 Avoiding Counterfeiting of Goods in Supply Chain with ZK Technologies

Validate that your goods have not been tampered with.
Preserve privacy.
Maintain reliability.

🔔 Real-time Notifications Tracker and Visualizer

Track specialized events on-chain
- Check your liquidation price on a Perp DEX.
- Identify new token launches
Build visualizers for on-chain metrics
- Inspiration from BlockClock Micro’s Visualizer for BTC metrics
- Showcase on-chain metrics in a surreal way.

🖼️ NFT Gateway Passes for events and special occasions

Real-life events with fraud-proof verification using on-chain NFTs.
Added layer of digital authenticity.
Savor the memorable experiences.

Accessibility for All

Kenshi’s vision is committed to increase the progress of blockchain data accessibility and bridging applications into the real-world, such that developers can interact in customizable interfaces and create seamless integrations to Web 3.0.

In the grand symphony of technology, from the birth of the Internet of Things to the evolution of blockchain innovation, one resounding theme emerges: the quest for data accessibility. As we journey through the IoT's historical awakening, witness the transformative power it brought to healthcare, agriculture, transportation, and logistics. And now Kenshi's vision is to stand at the cusp of a new era, one where IoT and blockchain unite to offer boundless possibilities.

In this ever-expanding landscape, data becomes the bridge between the tangible and the digital, the real and the decentralized. It's a bridge we can all cross, thanks to Kenshi's commitment to transparency, predictability, and scalability.

As we look ahead to Kenshi's tomorrow, imagine a world where IoT and blockchain seamlessly intertwine, where developers craft real-world applications effortlessly, and where data accessibility knows no bounds. The future is bright, and the possibilities are endless. It's a future where blockchain technology becomes a force for good, safeguarding supply chains, providing real-time insights, and adding a touch of digital authenticity to our most cherished moments.

So, whether you're navigating the complexities of supply chains, tracking real-time events, or preserving memories with NFT-centric events, Kenshi's data infrastructure empowers you to do so with ease. It's a future where data isn't a barrier but a bridge to innovation, and together, we're crossing it to amplify the impact of technology on our world.

Ecosystem Links

🧰 Build today on Kenshi: https://kenshi.io/dashboard
Discord: https://discord.gg/KenshiTech
🐤 Twitter: https://twitter.com/KenshiTech
📮 Telegram: https://t.me/KenshiTech
💻 Github: https://github.com/KenshiTech

Build Power Apps on a Budget with Kenshi Indexing

Pouya Eghbali — Mon, 04 Sep 2023 23:32:18 +0000

This article will delve into the exciting realm of blockchain smart contract indexing infrastructure and explore its pricing landscape. Discover the sheer power of Kenshi Deep Index - an exceptional solution meticulously crafted for seamless data consumption, cost-efficient, and efficient event management within the intricate world of EVM compatible blockchains.

Data Indexing at a Glance

Kenshi understood that the data infrastructure landscape has been growing tremendously, although with the remarkable level of accessibility of on-chain data, the costs associated with indexing new contracts for events on-chain has radically increased. We wanted to build infrastructure that would help facilitate and onboard all users who have the versatility to integrate their web apps, telegram/twitter/discord bots, and other infrastructure with the most ideal features.

Kenshi wanted to eliminate the friction of blockchain data development and empower users with unparalleled analytical capabilities that can enable insights into all sectors of on-chain applications, not limited to DeFi, NFTs, or LSDFi. Build tailored dashboards with smart contract functionalities across a multitude of EVM compatible chains.

Data Indexing Platforms Breakdown

Analytics Platforms

Many large data analytics providers have conveniently offered greater accessibility to on-chain data as a main stage for visualization. It illuminates opportunities for data scientists, engineers, and researchers to build insights with their technical experiences and share it to the public.

However, the initial hurdle of understanding how to build queries can be challenging for users integrating with the platform. Additionally, most notable platforms in this space were charging approximately $410k in credits for indexing usage with their API. This is in contrast to Kenshi's Reverse-API which costs $49.99 for the same service and allows for 2,592,000 indexing runs (or a run per second for the entire month).

Popular DEX Protocols

The emergence of DEX protocols has led to the creation of data infrastructure that integrates with platforms at a lower cost, while simulating environments that enable better understanding of retrieving on-chain events.
However, pricing for many of these protocols lacks transparency and requires payment on Ethereum, which incurs greater gas fees, making it more difficult for new developers to onboard. Kenshi takes a different approach by creating robust data infrastructure that is up-front, transparent, and cost-efficient. By being on Arbitrum, Kenshi spends less on gas fees (and even less with the introduction of EIP-4844) and offers competitive pricing.

How does Kenshi do it?

Kenshi's Deep Indexing ecosystem is a highly efficient system that periodically runs tasks, fetches events from the blockchain, and stores them into a distributed Kenshi blockchain data cluster across the world. This system is designed to ensure that all data is safely and securely stored, and can be accessed from anywhere in the world. One of the key benefits of this ecosystem is that syncing tasks is fully managed by our service, which means that you don't have to worry about setting up external servers to create your working environment.

Integration is made effortless too – thanks to our streamlined approach catered towards easy incorporation with existing systems involving smart contracts across multiple platforms on:

Ethereum
Arbitrum
Aurora
Avalanche
Bitgert
BNB Chain
Fantom
Polygon… and more

Enabling seamless integrations to your workflows without any hassle or disruption. Kenshi understands that navigating today's fast-paced digital economy, if you are looking for more EVM compatible chains, reach out to us here, Kenshi can integrate newly launched EVMs in 1-3 hours.

Indexing for All

Kenshi is a blockchain platform that is committed to solving the problem of unrealistic pricing models offered to developers in the blockchain ecosystems. The platform recognizes that accessibility for data infrastructure should not be opaque and indeterministic, and that developers need user-friendly interfaces that align perfectly to modern expectations.

To achieve its goal, Kenshi has developed an innovative pricing model that is transparent, predictable, and scalable. This model ensures that developers can easily access the resources they need to build and deploy their blockchain applications without having to worry about hidden fees or unexpected costs.

In addition, Kenshi's user-friendly interface is designed to simplify the development process and make it more accessible to a wider range of users. The platform's intuitive design enables developers to quickly and easily navigate through the various features and tools that they need to build their applications, while also providing them with the flexibility and customization options they need to meet their unique requirements.

The breadth of applications can open doors to revolutionize the way developers access and utilize blockchain technology. With its innovative pricing model and user-friendly interface, the platform is well-positioned to help drive and maximize the industry forward into the future with greater confidence.

Ecosystem Links

Clio: extremely fast, multi-threaded code on the browser

Pouya Eghbali — Mon, 20 Sep 2021 18:00:05 +0000

Clio is a functional programming language that compiles to JavaScript. It runs on Node.js as well as in the browser and is multi-threaded on both. Clio makes it super easy to do parallel or distributed programming, you can easily use the CPU cores on the local device, or use the resources on another device available on the network. Clio scales indefinitely. It has a clean syntax and is super fast. You can read an introduction to Clio here on medium, visit our GitHub repository or website, or you can directly go on to our playground to see and try a few examples.

Let me, however, show you a quick demonstration of Clio, and how to do multi-threading with it. To do multi-threading on the browser or on Node.js, all you need to do is to define your functions and call them using the |sandwich| syntax, Clio takes care of the rest:

If you want to, you can run this example directly on the playground.

In this example, there are a few little details that need explanation. On line 8 of the sample code above, you can see a demonstration of the Clio sandwich syntax |fib|, which runs the fib function in a separate thread. On the same line, you can see that -> * is used for mapping the data to the|fib|function. On line 9, (console.log @it) is an anonymous function that accepts it as its first parameter.

With Clio, it is also possible to import functions over the network and use them like any other regular function. Instead of making and implementing API endpoints, with Clio you can just host your functions and import them elsewhere, Clio gives you FaaS and micro-services architecture for free, without any extra hassle. For example

is valid Clio code. To see a demonstration of the remote functions, you can check out our todo example here.

The Clio v0.11.0 release, which just got recently published brings a couple of important changes. These changes are mostly performance-related: they improve the execution speed of Clio code, as well as improving the overall RPC and multi-threading performance. In this article, I’ll be showing you how fast Clio really is, and we will go through these new enhancements together. Let’s start with the performance!

How fast is Clio?

Since Clio compiles to JavaScript, everyone’s first assumption is that Clio is slower, or as slow as JavaScript. That might be true for a language like TypeScript, but Clio is actually faster than JavaScript, and in certain situations, it is extremely faster than JavaScript. I will explain how Clio can achieve such a performance later on in the article. But for now, let’s take a look at how fast we are talking about.

The above chart shows execution times for calculating the 1000th Fibonacci number in different programming languages. As you can see, Clio is almost as fast as C. Of course, I do not expect Clio to be this fast always or in every other situation, and I do not claim that Clio is as fast as C, what I claim is that Clio is super fast, and a lot of work has been done in making it fast. So how can it be so fast if it compiles to JavaScript?

First of all, I should mention that JavaScript, contradictory to the common belief, is actually a very fast programming language. Since Clio *compiles *to JavaScript, the compiler can apply compile-time optimizations to the generated JavaScript. In the case of the above example, tail call optimization is applied to the Fibonacci function. You can find the source code for the benchmark here.

Another reason why Clio runs faster than JavaScript is the way Clio code is written: using pipes, one can avoid creating intermediate constants or variables. The output of a function call can be directly piped to another function, and from there to another. Of course, it is also possible to nest function calls in JavaScript and avoid making intermediate variables, however, the Clio syntax was carefully designed to allow doing this while maintaining the readability. For example, the following JavaScript code:

Becomes this in Clio:

In the JavaScript example above, the order of execution of the functions is from the inside to the outside, but in the Clio code, it gets executed from left to right, in the exact same order you read and write the code. This makes a lot more sense if you think about it!

A lot of time and effort was spent on keeping Clio fast, and we had to sacrifice quite a few features along the way for that. As an example, we had to give up on laziness, because the V8 engine we run on doesn’t support it natively. However, we made sure that Clio compatible stays with JavaScript.

As a result of that, you can now import any JavaScript library into Clio and use it without facing any issues. As a result, you can even benefit from the multi-threading features of Clio to make your JavaScript libraries run faster:

In the above chart, you can see how parallelizing your tasks with Clio improves their performance! As an example, Jimp, a pure JavaScript image manipulation library was used to apply a greyscale effect to a directory full of images. With Clio, you can apply the effect to each image in a separate thread, which greatly improves performance. In Clio version v0.11.0 a lot of work was done to make parallelism even faster, more efficient and much better than ever before!

Parallelism in Clio

Clio has a network-based parallelism model. You can call it a distributed computing model. On startup, Clio creates a central hub or message broker, called a dispatcher, and listens to connections on different protocols based on the project’s configuration. Then it spawns workers and waits for them to connect to the dispatcher. A dispatcher is just a message hub or a broker that relays messages between the main thread and the workers. There is no shared memory between the workers and the main thread, so Clio has to serialize each and every message.

Prior to version v0.11.0, Clio used JSON for serialization. JSON is available everywhere and it’s fairly fast, so at first, it might sound like a good idea to go with JSON. Unfortunately, as soon as we start serializing custom types with JSON, its performance degrades to the point that it’s not efficient anymore. To solve this issue, I created a serialization library for JavaScript named Sia, and to boost its serialization performance to the next level, I also designed a new text encoding which I called utfz. I spent over a year optimizing these two libraries, and as a result, managed to make Sia so fast that the pure JavaScript version of it beats even the performance of the native serialization libraries available for Node.js!

Sia is a binary serialization format, and as you can see in the above chart, Sia is extremely fast! To see more benchmarks, or to learn more about Sia and how it was made so fast, you can refer to this medium article. Using Sia as a serialization library, Clio gained a big performance boost on multi-threading, its remote function calls and RPC protocol.

The above chart shows the performance gain of Clio after I made the switch to Sia. Not only it’s faster to serialize the data, it is now also faster to transfer the data. With Sia, the serialized data is much smaller than JSON and is already in binary format! On the browser also, the performance is greatly improved. For a quick demo, you can check the video below on how faster the Clio Mandelbrot example renders on Sia in comparison to JSON:

This video shows the render time for a multi-threaded Mandelbrot example written in Clio. You can check it in action here, or you can check the source code on GitHub. As you can see, the Sia version is much faster than the JSON version, and finishes almost in half the time as the JSON one!

With Clio, it’s super easy to run functions in a thread, no need to set up anything yourself, no dealing with the web workers or the worker threads, no need to implement communication protocols or synchronization, everything is already taken care of, just run clio new to scaffold a new project, write your code, then run the clio run command and you’ll get your app up and running in just a few seconds. You can check examples on the Clio examples repository.

Clio is still a very young language and it has a long way to go, it is not production-ready yet, and I’m not rushing for a stable release. I want to keep it minimal, and I want to carefully choose what goes in or what we should keep out of the language. It is designed with the cloud and micro-services in mind, and that’s where the main focus is. With Clio, I’m aiming for a general-purpose language that is usable both on and off the web, a language that is suitable for making a website, as well as crunching numbers and scientific data.

If you have any questions, leave a comment under this article, post an issue on GitHub, and if you want to talk, you’re welcome to join our public chat on Telegram!

clio-lang / clio

Clio is a functional, parallel, distributed programming language.

Clio

Clio is a functional, distributed programming language that compiles to JavaScript. It runs in the browser and on the server and is made to be a suitable language for scientific programming. Clio takes advantage of multiple CPUs and multiple CPU cores (parallelism) by default, is able to use network resources, is easy to run on clusters and on the cloud.

Read Clio documentation or visit our website for more info.

Install

To install you'll need Node.js (latest version) and NPM. Clio is hosted on NPM, to install it simply do

npm i -g clio

Command Line Usage

To see a list of available clio commands and their description you can run

clio --help

Project Status

Clio is in active development and it's not ready for production. It is in a highly experimental state, although some stable demonstrations and test programs exist, it is not recommended to use in production.

…

View on GitHub