DEV Community: Bijin Regi Panicker

Expert Iteration

Bijin Regi Panicker — Wed, 18 Sep 2024 12:49:35 +0000

EI is one of the methods under the umbrella of RL, which combines the best of supervised learning as well as reinforcement learning to make it possible for an agent to learn more rapidly. The basic concept behind EI is to bootstrap the learning of an agent through its process by first imitating some expert behavior and then enhancing it through reinforcement learning as well as self-play. This is an iterative process; that is, the agent can refine the policy incrementally by incorporating demonstrations from experts with its individual experience in the environment.

Expert Iteration can be considered to fundamentally focus on working with the demonstrations of an expert. Demonstrations from an expert refer to the sequences of actions implemented by an expert-a human or perhaps a well-trained agent-while in particular states of the environment. This type of demonstration gives the agent stronger indications of good actions in different contexts. The first stage of EI is called behavior cloning, wherein the agent uses supervised learning to imitate the behavior of an expert. This allows the agent to learn a baseline policy efficiently without exploitation of the environment or reliance on trial and error in pure reinforcement learning techniques like Q-learning or Proximal Policy Optimization (PPO).

After acquiring an initial policy from the expert demonstrations, the agency is left for self-play wherein the agent will play by itself using the reinforcements learned along the way to improve the policy. Under self-play as well as the aforementioned phase, an agent gets feedback from the environment -- which this time is in the form of rewards -- that enables it to explore the new strategies that will further promote its performance over that of the expert. Expert Iteration makes it possible for the agent to improve its policy over time as it alternates between learning from expert demonstrations and self-play, thus reaping the benefits of both the efficiency of supervised learning and the exploration benefits of reinforcement learning.

It significantly improves sample efficiency—the amount of data required to train an agent. Demonstrations by experts allow the agent to bypass the typically very time-consuming initial exploration phase of traditional RL methods, which is especially important in environments where exploration is costly or even dangerous (for example, robotics, or autonomous driving). For instance, EI allows progressive improvement because, through self-play, the agent can exploit it to improve over the expert's performance by finding new strategies not contained in the expert's data.

Conversely, Expert Iteration has a drawback. The quality of such an approach depends on good-quality demonstrations from the expert. Bad-quality or suboptimal demonstrations can nudge the agent to learn ineffectual or suboptimal policies. Furthermore, EI may not scale well in environments with large state and action spaces because high-quality expert demonstrations may take a very long time to collect and have very high costs. Yet another problem is that EI can potentially lead to overfitting of the agent's behavior on the expert's behavior, and thus restrain the agent from exploring alternative strategies during self-play.

Real applications of Expert Iteration have created tremendous evidence and successive applications to varying domains, like fast learning for manipulation or navigation tasks by robotics and game-playing AI systems like AlphaZero, demonstrating that self-play, together with expert knowledge, can lead to superhuman performance. More importantly, learning from expert drivers may be used as EI for bootstrapping an initial policy for training on its own to drive safely to then refine its behavior through reinforcement learning.

2024 Prediction

Bijin Regi Panicker — Thu, 28 Dec 2023 11:34:19 +0000

As we navigate the intricate web of technological advancements in 2024, the landscape is characterized by a convergence of transformative trends across various sectors. Beyond the well-established pillars of 5G, artificial intelligence (AI), and digital wallets, a myriad of factors ranging from space exploration to quantum computing and augmented reality are set to reshape the future.

The anticipated global adoption of 5G technology is a linchpin for innovation. By the end of 2024, 60% of Communications Service Providers (CSPs) are expected to commercialize 5G services, covering Tier-1 cities worldwide. The proliferation of 5G networks is forecasted to reach 352, ushering in a new era of high-speed connectivity and enabling a multitude of applications across industries.

In the realm of healthcare, AI is not merely a trend but a fundamental shift in how the industry operates. The average estimated budget allocation to AI/ML technologies is projected to grow from 7% in 2022 to 15% in 2024. AI and the cloud emerge as a potent combination that is set to dominate in 2024, revolutionizing diagnostics, treatment planning, and overall patient care.

Digital wallets, driven by mobile commerce, are poised to become the most popular online payment method globally by 2024. Accounting for over a third of all payments, this shift reflects the increasing preference for convenient and secure digital transactions in the era of e-commerce dominance.

Autonomous mobility is advancing rapidly, with Level 4 (L4) highway pilots becoming possible by 2024 or 2025 for private cars. Trucking is expected to be among the earliest use cases for autonomy in the commercial segment, indicating a transformative shift in transportation.

Next-generation cloud technologies are set to witness substantial innovations in 2024. Customers are likely to opt for private cloud infrastructures and focus on equipment deployment at the edge. This reflects the industry's need to manage complex hybrid cloud environments more efficiently, ensuring scalability and flexibility.

Smart contracts and blockchain are central to the evolution of digital transactions. By 2024, blockchain technology is expected to be in the top five strategic priorities for many businesses. The focus on security, scalability, and legal issues in blockchain-enabled smart contracts signals a maturation of the technology, making it a cornerstone of digital innovation.

Space exploration and reusable rockets are at the forefront of scientific endeavors. NASA's plan to return to the Moon in 2024 marks a significant milestone, accompanied by a plethora of private lunar landers, probes to Venus and Jupiter, and reusable heavy-lift rocket tests. The private sector, exemplified by SpaceX and Rocket Lab USA, is driving innovation in space travel.

In the realm of biology and healthcare, programmable biology is emerging as a transformative field. The Generative and Synthetic Genomics Programme at the Wellcome Sanger Institute is pioneering efforts to predict and engineer biology. Synthetic biology, with trends like CRISPR-Cas9 gene editing and automated DNA synthesis, promises groundbreaking advancements.

Precision therapies, particularly in the cell therapy market, are expected to surge, reaching a predicted forecast of over $52 billion by 2029. Precision medicine trials, often oncology-focused, are leveraging the mechanisms of action (MoA) of candidates to target and kill T-cells.

The 3D printing revolution is poised to enter a new phase in 2024, moving beyond prototyping and niche applications. The technology's increased affordability and accessibility will facilitate its integration into various industries, from healthcare and automotive to construction and consumer goods.

Adaptive robotics, despite facing economic challenges and high-interest rates in 2024, is projected to witness significant growth. The World Robotics Report anticipates robot installations to reach nearly 600,000, with generative AI emerging as a transformative force within robotics.

Cryptocurrencies continue to capture the financial landscape, with Bitcoin predicted to reach a value of $100,000 in 2024. Ethereum is also expected to experience significant growth. Tokenized alternative assets are gaining traction, offering investors diverse opportunities beyond traditional financial instruments.

Intelligent devices and the Internet of Things (IoT) are set to shape the future of connectivity. The IoT industry is projected to be worth over $1 trillion by 2024, with the smart home sector leading in deploying IoT devices. Advancements in machine learning, artificial intelligence, and the rise of smart cities underscore the limitless possibilities for IoT applications.

Neural networks and AI applications are expanding their reach, creating realistic text, images, and music while automating algorithm improvements. In 2024, more sophisticated AI applications and algorithms are expected, optimizing data, performing complex tasks, and making decisions with human-like accuracy.

As we delve into the realm of cybersecurity, AI is projected to play a crucial role in tracking and monitoring emissions in the climate tech landscape. The evolution of AI to become more broadly accessible while addressing the reliability, diversity, and privacy of data underscores its dual role as both an attack tool and a target.

The renewable energy industry anticipates an 8% growth in global energy consumption in 2024. Despite challenges such as high prices and supply chain disruptions, demand for renewable energy is set to rise by 11%. The

Energy Information Administration expects renewable deployment to grow by 17%, accounting for almost a quarter of electricity generation.

Blockchain and DeFi (Decentralized Finance) continue to be strategic priorities for businesses. In 2024, the rise of Layer 2 solutions is poised to scale DeFi, reduce transaction costs, and make it even more attractive to both users and institutional investors.

In the realm of b and sustainability, AI is expected to play a crucial role in tracking emissions and monitoring them. McKinsey's upcoming report on climate technologies is set to highlight the potential of various technology categories, emphasizing collaboration between startups, corporations, and governments for holistic climate solutions.

Global economic trends project a 7% growth in the global economy in 2023 and 2024, with the global economy expected to outperform expectations. Goldman Sachs Research anticipates a positive economic performance, mirroring the resilience observed in 2023.

Quantum computing is on the brink of a new era, transitioning from physical qubits to error-corrected logical qubits. Increased global collaboration in quantum research signifies the industry's focus on addressing the critical issue of error correction and practical problem-solving applications.

Augmented Reality (AR) and Virtual Reality (VR) are poised to generate significant revenue, with the market projected to reach $1 billion by 2024. The annual growth rate of 60% from 2023 to 2028 underlines the increasing adoption of AR and VR technologies across industries.

Customize Programming Languages

Bijin Regi Panicker — Thu, 15 Jul 2021 13:22:20 +0000

This post is to introduce LangTrans.

LangTrans is preprocessor to convert code written in your syntax into code in original syntax.

You can create syntax for

your own needs
project needs
your domain

No need to know about inner workings of compiler.
Knowledge about regular expression is only prerequisite.

LangTrans / LangTrans

Customize programming languages

To customize any programming language

LangTrans is a syntactic preprocessor
It helps you to customize the syntax of any programming language
It converts customized syntax to original syntax.
It uses regular expression but it supports nesting(called part calling).

Example

Customized Syntax of Python

#Print
p"Hello World"
# Anonymous function
inc = (x) => x+1
# Lambda function
twice(x) = 2*x
# Single Line try-except
try inc("1") Exception print("Error:",err)
# Print Done if x is defined other wise Failed
print((x||True)?"Done":"Failed")
# Single Line if and check x defined or not
print('x is not defined') if !x
# Pipe Syntax
1 -> inc
|> print
# Arithmetic operations with functions 
print((inc+twice)(3))

…

View on GitHub