DEV Community: Iurii Panarin

Your LLM can scrape the web — locally, without writing throwaway code

Iurii Panarin — Thu, 23 Jul 2026 18:49:12 +0000

Every time you ask an AI assistant to "get data from this website", one of two things happens: it hallucinates the data, or it writes a one-off Python script you'll never run again. Both are wasteful. The model already knows where the data lives and how to select it — what's missing is a safe, reusable way to execute that knowledge.

That's what fitter does. It's a small Go engine with one idea at its core: web extraction should be a config, not code. A fitter config declares where the data lives (HTTP request, headless browser, static value, file) and what to extract (gjson paths, CSS selectors, XPath). The engine does the rest.

And because configs are plain JSON/YAML, an LLM can author them. Fitter ships an MCP server, so Claude Code, Claude Desktop, or any MCP client can write a config, validate it, and run it on your machine — in one conversation turn.

60-second setup

Grab a binary from the releases page (or go build -o fitter_mcp ./cmd/mcp), then:

claude mcp add fitter -s user -- /path/to/fitter_mcp

Claude Desktop users: download the .mcpb bundle for your platform from the same release page and open it — that's the whole install.

The server exposes five tools: fitter_run, fitter_run_file, fitter_run_url, fitter_validate_config, and fitter_config_reference — that last one returns a condensed format reference, so the model can teach itself the config format without leaving the conversation.

"Get me the latest Hacker News stories"

Ask exactly that. Behind the scenes, the model authors something like this and calls fitter_run:

{
  "item": {
    "connector_config": {
      "response_type": "json",
      "url": "https://hacker-news.firebaseio.com/v0/newstories.json",
      "server_config": { "method": "GET" }
    },
    "model": {
      "array_config": {
        "root_path": "@this",
        "length_limit": 10,
        "item_config": {
          "field": {
            "type": "object",
            "generated": { "model": {
              "type": "object",
              "connector_config": {
                "response_type": "json",
                "url": "https://hacker-news.firebaseio.com/v0/item/{PL}.json",
                "server_config": { "method": "GET" }
              },
              "model": { "object_config": { "fields": {
                "title": { "base_field": { "type": "string", "path": "title" } },
                "by":    { "base_field": { "type": "string", "path": "by" } },
                "score": { "base_field": { "type": "int",    "path": "score" } },
                "url":   { "base_field": { "type": "string", "path": "url" } }
              } } }
            } }
          }
        }
      }
    }
  },
  "limits": { "host_request_limiter": { "hacker-news.firebaseio.com": 5 } }
}

Read it top to bottom: fetch the ID list, take the first 10, and for each element fire a nested request ({PL} is the current value) to fetch the story object — rate-limited to 5 parallel requests against the HN API. The result is clean JSON:

[
  {"title": "Fields Medals 2026", "by": "nill0", "score": 2, "url": "https://www.mathunion.org/..."},
  {"title": "Writing by Hand is Good for your Brain", "by": "dwwoelfel", "score": 2, "url": "..."}
]

Here's the part that matters: that config is an artifact, not an afterthought. Save it to a file and it's a CLI command, a cron job, or a service — the model's one-off answer became infrastructure.

One config, four data sources

My actual morning briefing — weather, HN, world headlines, and the Bitcoin price — is one config with four independent branches. It also shows off placeholders:

"connector_config": {
  "response_type": "json",
  "url": "https://wttr.in/{{{FromInput=.}}}?format=j1"
}

{{{FromInput=.}}} is filled from the tool's input argument at run time, so "run my briefing for Berlin" and "for Paris" reuse the same config. There are placeholders for env vars, cached references (think: fetch a JWT once, use it in every request's headers), array indices, and more.

Join sources: scrape what has no API, enrich from the one that does

GitHub's trending page famously has no API. But the repos on it do. So: scrape the HTML for the repo slugs, then fan each one out into the real GitHub REST API:

{
  "item": {
    "connector_config": {
      "response_type": "HTML",
      "url": "https://github.com/trending",
      "server_config": {
        "method": "GET",
        "headers": { "User-Agent": "Mozilla/5.0 (fitter demo)" }
      }
    },
    "model": {
      "array_config": {
        "root_path": "article.Box-row h2 a",
        "length_limit": 5,
        "item_config": {
          "field": {
            "type": "string",
            "html_attribute": "href",
            "generated": { "model": {
              "type": "object",
              "connector_config": {
                "response_type": "json",
                "url": "https://api.github.com/repos{PL}",
                "server_config": { "method": "GET", "headers": { "User-Agent": "fitter-demo" } },
                "null_on_error": true
              },
              "model": { "object_config": { "fields": {
                "repo":        { "base_field": { "type": "string", "path": "full_name" } },
                "stars":       { "base_field": { "type": "int", "path": "stargazers_count" } },
                "language":    { "base_field": { "type": "string", "path": "language" } },
                "open_issues": { "base_field": { "type": "int", "path": "open_issues_count" } },
                "description": { "base_field": { "type": "string", "path": "description" } }
              } } }
            } }
          }
        }
      }
    }
  },
  "limits": { "host_request_limiter": { "api.github.com": 2 } }
}

Today's actual output:

[
  {"repo": "block/buzz", "stars": 6214, "language": "Rust", "open_issues": 460,
   "description": "A hive mind communication platform"},
  {"repo": "koala73/worldmonitor", "stars": 71179, "language": "TypeScript", "open_issues": 272,
   "description": "Real-time global intelligence dashboard. AI-powered news aggregation..."},
  {"repo": "shiyu-coder/Kronos", "stars": 32946, "language": "Python", "open_issues": 241,
   "description": "Kronos: A Foundation Model for the Language of Financial Markets"},
  {"repo": "Pumpkin-MC/Pumpkin", "stars": 8808, "language": "Rust", "open_issues": 207,
   "description": "Empowering everyone to host fast and efficient Minecraft servers."},
  {"repo": "citrolabs/ego-lite", "stars": 1485, "language": "JavaScript", "open_issues": 18,
   "description": "The best browser for both you and your AI agents work in parallel."}
]

The details worth stealing:

HTML and JSON mix freely. The outer connector parses HTML with a CSS selector (article.Box-row h2 a); each matched element fans out into a JSON API request. One config, two worlds.
html_attribute reads attributes, not text. The anchor's href is already /owner/repo — which is exactly what https://api.github.com/repos{PL} needs. The join key comes from the page itself, so it can't drift out of sync.
length_limit + host_request_limiter keep you polite. Top 5 repos, max 2 concurrent API calls. Unauthenticated GitHub API allows 60 requests/hour — a config that respects that is a config you can run on a schedule.

This is the general pattern for the modern web: the interesting ranking lives in some HTML page, the facts live in an API, and the join between them is exactly one {PL} away.

When the join key is a field, not the whole value

{PL} injects the entire current value — perfect when your array items are scalars (story IDs, repo slugs). But often the items are objects and the join key lives inside them. That's what expression placeholders are for. Book search → author enrichment on OpenLibrary:

{
  "item": {
    "connector_config": {
      "response_type": "json",
      "url": "https://openlibrary.org/search.json?q={{{FromInput=.}}}&limit=4&fields=title,author_key,first_publish_year",
      "server_config": { "method": "GET" }
    },
    "model": {
      "array_config": {
        "root_path": "docs",
        "item_config": {
          "fields": {
            "title": { "base_field": { "type": "string", "path": "title" } },
            "year":  { "base_field": { "type": "int", "path": "first_publish_year" } },
            "author": {
              "base_field": {
                "type": "object",
                "generated": { "model": {
                  "type": "object",
                  "connector_config": {
                    "response_type": "json",
                    "url": "https://openlibrary.org/authors/{{{FromExp=fromJSON(fRes).author_key[0]}}}.json",
                    "server_config": { "method": "GET" },
                    "null_on_error": true
                  },
                  "model": { "object_config": { "fields": {
                    "name": { "base_field": { "type": "string", "path": "name" } },
                    "born": { "base_field": { "type": "string", "path": "birth_date" } },
                    "died": { "base_field": { "type": "string", "path": "death_date" } }
                  } } }
                } }
              }
            }
          }
        }
      }
    }
  },
  "limits": { "host_request_limiter": { "openlibrary.org": 2 } }
}

Run with input = dune:

[
  {"title": "Dune",             "year": 1965,
   "author": {"name": "Frank Herbert", "born": "8 October 1920", "died": "11 February 1986"}},
  {"title": "Dune Messiah",     "year": 1969,
   "author": {"name": "Frank Herbert", "born": "8 October 1920", "died": "11 February 1986"}},
  {"title": "Children of Dune", "year": 1976,
   "author": {"name": "Frank Herbert", "born": "8 October 1920", "died": "11 February 1986"}}
]

The key line is the nested URL: {{{FromExp=fromJSON(fRes).author_key[0]}}} is an expr-lang expression evaluated against the current item (fRes) — parse it, take the first author key, splice it into the URL. Anything expr-lang can compute can become part of a request: pick a field, concatenate, add an offset, branch on a condition. And notice the output shape: each book keeps its own scalar fields while the enrichment lands as a nested author object — the config's structure is the response's structure.

Don't just return data — write the report

Sometimes the deliverable isn't JSON in a chat window, it's a file on disk. The file_storage generated field turns any field into a write: the top-5 crypto coins, straight into a CSV:

{
  "item": {
    "connector_config": {
      "response_type": "json",
      "url": "https://api.coingecko.com/api/v3/coins/markets?vs_currency=usd&order=market_cap_desc&per_page=5&page=1",
      "server_config": { "method": "GET" }
    },
    "model": {
      "array_config": {
        "root_path": "@this",
        "item_config": {
          "field": {
            "type": "object",
            "generated": {
              "file_storage": {
                "content": "{HUMAN_INDEX},{{{name}}},{{{current_price}}},{{{price_change_percentage_24h}}}\n",
                "file_name": "coins.csv",
                "path": "/tmp/fitter-report",
                "append": true
              }
            }
          }
        }
      }
    }
  }
}

The tool call returns the written paths, and on disk:

$ sort -n /tmp/fitter-report/coins.csv
1,Bitcoin,64778,-2.3
2,Ethereum,1881.01,-3.4
3,Tether,0.999265,0
4,BNB,566.81,-1.6
5,USDC,0.999672,0

What's going on in that content template:

Bare {{{json.path}}} placeholders pull fields straight out of the current item — {{{name}}}, {{{current_price}}} — no expressions needed for simple lookups. (Type the field as object so the template sees the item's raw JSON.)
{HUMAN_INDEX} stamps the item's position (1-based; {INDEX} is the 0-based sibling). Fitter processes array items concurrently, so appends land in completion order — the rank column is what makes sort -n restore truth. A nice reminder that this thing is genuinely parallel.
There's a sibling file field type that downloads — point it at an image/PDF URL and the field's value becomes the local file path. Scrape a gallery, keep the pictures.

Combined with schedule-based service mode, this is a tiny ETL pipeline: fetch → shape → append to a growing CSV, every hour, defined entirely in one JSON document.

When the site has no API

The examples above hit JSON APIs, but the response_type can just as well be HTML (CSS selectors), xpath, or XML — and the connector can be a real browser when the page needs JavaScript:

"connector_config": {
  "response_type": "HTML",
  "url": "https://example.com/spa-page",
  "browser_config": { "playwright": { "browser": "Chromium", "install": true, "stealth": true } }
}

Chromium, Playwright (with stealth), and dockerized browsers are all built in — one static binary, no npm install.

Why local-first matters

The hosted scraping APIs are good products, but for agent workflows the local model has real advantages:

Auditable — you can read exactly what the agent fetched and how. A config diff is reviewable; generated code in a sandbox somewhere is not.
No keys, no metering — your machine makes the requests. Nothing to sign up for, no per-page billing, no data passing through a third party.
Reusable by construction — the same config runs via MCP, CLI (fitter_cli --path config.json), a Go library call, or fitter's service mode with schedulers and notifiers (Telegram, webhook, Redis, file). Authored once by the model, promoted to a cron job when it proves useful.

Hosting it for your team

New in the latest release: the MCP server also speaks streamable HTTP, so you can run one fitter for the whole team. A minimal docker-compose.yml:

services:
  fitter-mcp:
    image: ghcr.io/pxyup/fitter-mcp:latest
    restart: unless-stopped
    ports:
      - "8080:8080"
    environment:
      FITTER_MCP_HTTP_ADDR: ":8080"
      FITTER_MCP_AUTH_TOKEN: "${FITTER_MCP_AUTH_TOKEN:?set a token}"
    volumes:
      # optional: mount reusable configs for fitter_run_file
      - ./configs:/configs:ro

export FITTER_MCP_AUTH_TOKEN=$(openssl rand -hex 16)
docker compose up -d
curl http://localhost:8080/healthz   # -> ok

Every /mcp request must then carry the bearer token, and the image's built-in healthcheck keeps your orchestrator honest. Register the endpoint in Claude Code:

claude mcp add --transport http fitter http://your-host:8080/mcp \
  --header "Authorization: Bearer $FITTER_MCP_AUTH_TOKEN"

The image is slim by design (binary + CA certs, linux/amd64 + arm64): API/HTML scraping works out of the box, while browser-rendered pages need a release binary on a host with Chromium or Playwright.

Client-side cancellation propagates all the way down to in-flight HTTP requests and browser sessions, so an abandoned agent run doesn't keep scraping in the background.

Try it

Repo: github.com/PxyUp/fitter (MIT)
MCP registry: io.github.PxyUp/fitter
Ready-made configs: examples/

Open Claude Code, add the server, and ask for some data you actually want every morning. The config it writes is yours to keep — that's the point.

Questions and config contributions welcome — the examples folder takes PRs.

Fitter - Open Source no-code tool for map-reduce data from different source and even more!

Iurii Panarin — Wed, 30 Oct 2024 18:46:10 +0000

Hello everyone!

I am Pxyup and today i want represent for you by open source project Fitter.

PxyUp / fitter

New way for collect information from the API's/Websites

Fitter + Fitter CLI

Fitter - new way for collect information from the API's/Websites

Fitter CLI - small cli command which provide result from Fitter for test/debug/home usage

Fitter Lib - library which provide functional of fitter CLI as a library

Way to collect information

Server - parsing response from some API's or http request(usage of http.Client)
Browser - emulate real browser using chromium + docker + playwright/cypress and get DOM information
Static - parsing static string as data

Format which can be parsed

JSON - parsing JSON to get specific information
XML - parsing xml tree to get specific information
HTML - parsing dom tree to get specific information
XPath - parsing dom tree to get specific information but by xpath

Use like a library

go get github.com/PxyUp/fitter

package main
import (
    "fmt"
    "github.com/PxyUp/fitter/lib"
    "github.com/PxyUp/fitter/pkg/config"
    "log"
    "net/http"
)

func main() {
    res, err := lib.Parse

…

View on GitHub

How it was created

In 2023, I worked on an idea called Trip Searcher:

1.  You enter a budget.
2.  You specify a starting city or country.
3.  You set the trip duration and a range of possible start and end dates.

The Trip Searcher would monitor flights and return potential routes from the starting city, including total prices to various destinations, and send notifications to Telegram with:

1.  Flight costs (parsed from Google or Kiwi).
2.  Hotel prices (from Airbnb or Booking).
3.  Food costs (retrieved from Numbeo).

For this setup, I needed a list of countries, cities, and airport codes to plug into the sites mentioned. During development, I started thinking about how convenient it would be if all this information could be easily combined and parsed to streamline requests, which led to the idea for a project I call Fitter.

P.S.: This project was for personal use.

Fitter CLI

A no-code map-reducer that returns data in user-friendly (JSON) or custom formats, suitable for storage in a database or transmission via HTTP.

Features:

Supports parsing through HTML (query), JSON (gjson), XML, and xpath parsers.
Retrieves data as a browser would, using Docker, Playwright(+ stealth mode), HTTP Client, Cache, File, or propagated fields, with support for custom plugins.
Provides proxy support for Playwright and HTTP clients.
Can send or store information to a file, webhook, console, and more via plugins.
Handles all data types: int, float, bool, array, object, null, and string.
Combines (map-reduce) and transforms fields.
Utilizes the powerful expr library for template syntax, which is available across the application.
Offered as a standalone binary and Docker version.
Allows limits on request counts or instances for browser/Docker usage.

Examples

Static generation:

Here we will just generate static array from hardcoded

./fitter_cli_v1.0.18-darwin-amd64 --url=https://raw.githubusercontent.com/PxyUp/fitter/refs/heads/master/examples/cli/config_static_connector.json

[
        "PAGE: 1 INDEX: 0",
        "PAGE: 2 INDEX: 1",
        "PAGE: 3 INDEX: 2",
        "PAGE: 4 INDEX: 3",
        "PAGE: 5 INDEX: 4"
]

Get current time

Get information from the website and return to user.

./fitter_cli_v1.0.18-darwin-amd64 --url=https://raw.githubusercontent.com/PxyUp/fitter/refs/heads/master/examples/cli/config_current_time.json

"Current time is: 19:18:51"

Get current Steam Sales

That will create sales.md wile in provided directory

Get best news from HackerNews + Comment list for each

In this config we using template syntax for propagate result from the first request to next one.

./fitter_cli_v1.0.18-darwin-amd64 --url=https://raw.githubusercontent.com/PxyUp/fitter/refs/heads/master/examples/cli/config_cli.json

{
  "internal_url": "https://news.ycombinator.com/item?id=41975047",
  "content": {
    "kids": [
      {
        "response_id": "9db6ec73-3e35-4d06-b5eb-dbaa05a3a31d",
        "internal_url": "https://news.ycombinator.com/item?id=41975981",
        "content": {
          "text": "&amp;gt; We describe Flock as &amp;quot;Flutter+&amp;quot;. In other words, we do not want, or intend, to fork the Flutter community. Flock will remain constantly up to date with Flutter.&lt;p&gt;That was the first fear when I saw the title - splitting community and having two incompatible versions. Good to see it addressed in the post.&lt;p&gt;The second was just a fear of how it would complicate the development process, but it seems to be a drop-in replacement (just configuring FVM - Flutter Version Manager):&lt;p&gt;&lt;pre&gt;&lt;code&gt;   Configure .fvmrc to use Flock:   {     &amp;quot;flutter&amp;quot;: &amp;quot;master&amp;quot;,     &amp;quot;flutterUrl&amp;quot;: &amp;quot;https:&amp;#x2F;&amp;#x2F;github.com&amp;#x2F;Flutter-Foundation&amp;#x2F;flutter.git&amp;quot;   }&lt;/code&gt;&lt;/pre&gt;Flutter is the best thing that happened to UI development since Qt. Most people don&amp;#x27;t realize how many apps written in Flutter they use daily, simply because it&amp;#x27;s impossible to tell. And the frustration described in the post is felt by many CTOs and developers. Especially those who use Flutter for desktop and web. Flutter provides an amazing experience for desktop apps, and precisely because of that, it feels so frustrating when you stumble upon some stupid bug that has been open for a year or two and never gets prioritized. Usually, it&amp;#x27;s nothing critical, but still requires workarounds and wasting time.&lt;p&gt;I don&amp;#x27;t know, the idea of Flock sounds good, the main question is engaging the community. Hopefully, the author (who seem to be an ex-Flutter team member himself) have a good grasp on the state of the community.&lt;p&gt;Wishing luck to the project and going to keep an eye on the progress.",
          "title": "comment"
        },
        "id": 41975981
      },
      {
        "content": {
          "text": "Back when I worked on GWT, we had trouble accepting outside contributions because the team had a mandate to support Googlers. That is, much like other libraries and tools at Google, changes could not break google3. This means &lt;i&gt;testing&lt;/i&gt; patches against google3 and either changing the patch, or fixing whatever code used it, and these are tasks that no outsider can do.&lt;p&gt;Shepherding these patches is no fun when you have your own changes to work on that are more important to the team.&lt;p&gt;We did something similar, by creating an external fork where changes could be tried out by the community, without necessarily being accepted into the internal version.&lt;p&gt;I think a fork &lt;i&gt;could&lt;/i&gt; work if there was enough external momentum, but even 20 people working full time would actually be pretty good for an open source project. How many developers will this fork attract? The fork would need to attract other businesses who can put people on it.&lt;p&gt;One downside is that the code isn&amp;#x27;t tested against google3. Sometimes you find actual bugs that way.&lt;p&gt;Edit: reading more closely, the complaint doesn&amp;#x27;t seem to be that patches weren&amp;#x27;t reviewed, but rather that bug reports weren&amp;#x27;t investigated. That&amp;#x27;s definitely something outside developers could do more of, and seems a lot easier than forking?",
          "title": "comment"
        },
        "response_id": "af15ca55-c45c-48a8-a18d-2fcbd7dd8f3b",
        "id": 41975765,
        "internal_url": "https://news.ycombinator.com/item?id=41975765"
      }
    ]
  }
}

Scrape all images from website and store them locally

./fitter_cli_v1.0.18-darwin-amd64 --url=https://raw.githubusercontent.com/PxyUp/fitter/refs/heads/master/examples/cli/config_image_multiple.json

[
        "/Users/pxyup/fitter/bin/1/basic-image.png",
        "/Users/pxyup/fitter/bin/2/alt-text.png",
        "/Users/pxyup/fitter/bin/3/no-size.png",
        "/Users/pxyup/fitter/bin/4/size.png",
        "/Users/pxyup/fitter/bin/5/image-with-title.png"
]

Fitter

Fitter it is extended version of the Fitter CLI which have:

Http server for trigger
Return response as telegram message/webhook
And that currently did not have documentation :)

Usage

This tools can be used in different purpose:

Web scrapper
Data scrapper with plugins
Produce specific load testing
Build chat bots - i use it for automate my telegram channel

For example this job every day send best Dev.to aritcles:

Plans

Add more browser tools. Like click/scroll(currently can be done only by JS injection)
Improve template syntax
Add custom template editor + config editor
May be will think about SASS for fitter CLI for run custom workflow for customers and return result to APP/Watch/etc.

Really wait for your feedback! Ask any question i will ask

Fitter - open-source instrument for amazing scraping.

Iurii Panarin — Tue, 30 Jan 2024 22:46:18 +0000

Hello everyone!

I want to introduce Fitter for you.

PxyUp / fitter

New way for collect information from the API's/Websites

Fitter + Fitter CLI

Fitter - new way for collect information from the API's/Websites

Fitter CLI - small cli command which provide result from Fitter for test/debug/home usage

Fitter Lib - library which provide functional of fitter CLI as a library

Way to collect information

Server - parsing response from some API's or http request(usage of http.Client)
Browser - emulate real browser using chromium + docker + playwright/cypress and get DOM information
Static - parsing static string as data

Format which can be parsed

JSON - parsing JSON to get specific information
XML - parsing xml tree to get specific information
HTML - parsing dom tree to get specific information
XPath - parsing dom tree to get specific information but by xpath

Use like a library

go get github.com/PxyUp/fitter

package main
import (
    "fmt"
    "github.com/PxyUp/fitter/lib"
    "github.com/PxyUp/fitter/pkg/config"
    "log"
    "net/http"
)

func main() {
    res, err := lib.Parse

…

View on GitHub

This tool can do a lot of stuff:

Read content by XML/XPath/Json/HTML selectors
Emulate browser with docker/playwright/chromium
Support proxy
Global references
Fallback on no empty fields
Notification Telegram/Http/Redis/Console
File downloading
Full README.md
Can work as golang library
Support runtime golang plugins
Work as CLI
Works on Linux/Mac/Windows.
Support different type of fields
Support expression language for field and notification settings

......

N+1. Best and most important it is gluing of request

Repository have a multiple examples for CLI/Fitter

Real world example of the problem

Last week my friend need to monitor pool on some crypto stock. So here his problem:

First of all you need get all user positions https://docs.saucerswap.finance/v/developer/saucerswap-v2/liquidity-operations/get-user-positions
After that all stock pool https://docs.saucerswap.finance/v/developer/saucerswap-v2/liquidity-operations/fetch-all-pools
After that you should check if pool has user positions you should send notification.

Let's see how Fitter can solve this problem:

Let me explain what is here:

environment variables:

TELEGRAM_TOKEN - line 95

USER_ID - line 37

So my friend need just run Fitter and add UserIds for Telegram notification(line 95):

TELEGRAM_TOKEN=xxx USER_ID=xxx ./fitter --path=./config.json

What happens in config:

2.1 Creating Reference(line 3) on all pools and caching on 60 seconds(line 4). We take the response and create an array of the objects with two fields tokenA and tokenB.

2.2 Creating Reference(line 33) on AccountId will be just string from ENV.

2.3 We create items for monitoring every 10 seconds(line 88) which take values from the server via injecting AccountIdRef(line 52). Response it is array of objects with two field: first it is (line 66) - template string with injecting token name; second is boolean field which show if pair in pool(line 71)

Next we will inform if some of the values in array is true(line 93).

We will receive notification in Telegram:

[
 {
  "position": "tokenA=USDC tokenB=USDC[hts]",
  "value": true
 },
 {
  "value": false,
  "position": "tokenA=USDC tokenB=USDT"
 }
]

SafeStorageBot - keep’s your secrets in Telegram

Iurii Panarin — Fri, 20 May 2022 15:42:09 +0000

The Telegram team announced the Saved Messages tab quite a while ago. Over time, my friends and I noticed that some of our passwords from services began to appear there as well, and “Saved Messages” is already being used as a password repository.

It’s quite convenient, but there are some disadvantages:

Passwords are not encrypted.
It is hard to search without tags, because there is other information
It’s inconvenient to associate the service with the password
You have to select when copying (which is not always convenient when there are special characters)

Solve this problem securely for passwords with a bot, before the release of Telegram WebApp was impossible with one bot(you can encrypt separately and send the result):

No client side(code that can be verified)
Send unencrypted passwords to server is unsafe

With the advantage of WebApp this can be done (the main thing is to think up and remember the master-password):

Encrypt the password with a master password using Javascript (crypto API/crypto-JS) on the client side
Send the encrypted password to the server (there we encrypt the password a couple more times to secure the database)
When we need a password, we request it and decrypt it using the master password on the client side

That’s how this bot came about.

BOT LINK

Below I will show a short video of the bot:

How it works:

You add a password with the title
You enter the master password for encryption
Send title + encrypted password to the server (master-password is never sent)
On the server, encrypt the password and userId a second time with the server master-password
The entry is saved in the database ……
You request the password
The entry is retrieved from the database and decrypted with the server master-password
Sent to the client side
You enter your master password and decrypt on the client side using javascript
You see the decrypted password and can copy it

The client part of the application is posted in the repository:

SafeStorageBot / frontend_bot

Frontend for bot

TelegramPasswords

This project was generated with Angular CLI version 13.3.5.

Development server

Run ng serve for a dev server. Navigate to http://localhost:4200/. The application will automatically reload if you change any of the source files.

Code scaffolding

Build

Run ng build to build the project. The build artifacts will be stored in the dist/ directory.

Running unit tests

Run ng test to execute the unit tests via Karma.

Running end-to-end tests

Run ng e2e to execute the end-to-end tests via a platform of your choice. To use this command, you need to first add a package that implements end-to-end testing capabilities.

Further help

To get more help on the Angular CLI use ng help or go check out the Angular CLI Overview and Command Reference page.

View on GitHub

With the command /version you can always see the commit number of the client part of the application and see the code.

Frontend encryption module

I have long thought about monetisation and after consulting with friends I settled on the option 3 passwords can always be stored, and for a donation of 3 euros a month you can store any number

Thank you for your attention!

PS: Let me take advantage of the opportunity to be promoted again:
BOT LINK

How I built a super fast JS/TS framework

Iurii Panarin — Fri, 26 Apr 2019 14:52:46 +0000

How do other frameworks operate?

What if we didn't use Virtual DOM at all, instead work with real DOM instead? To answer this, we first need to understand how most frameworks work, to better illustrate this I drew a small example.

Violet are DOM/VDOM nodes, Blue = state, Red = diffing, Green = patching. I should explain what happens in this picture. So most of the frameworks that have state and DOM, act like this when something in the state is modified. In this example, state.bg has changed. So now we iterate through all of the nodes (those 5 steps) and renders them. If something is changed, then patching happens where rendered stuff goes to actual DOM (green action). Ok, that's cool, but why do we need to iterate every single node (note that here come those diffing algorithms that compete with each other and some does diff differently) when only state.bg changes?!..

So back to my original question “Why do we need Virtual DOM ?”, the answer is to iterate through all of DOM more quickly. Ok, so what if we didn't iterate?

Utopia way of doing this

But I want to show you the way that in my head it should work without diffing nor iterating, nor Vnodes for that matter as well.

So in this utopia way we changed the same state.bg and there is no iteration (so no need for Virtual DOM), only one render (that single red dot), only one single parameter/attribute/node modifies which is affected by state.bg changes. Much less unnecessary iteration, unnecessary memory usage for virtual nodes, much less rendering = much faster.

Motivation

1. Small bundle size
Many of the modern frameworks are very large. I honestly do not believe that the application “Hello World” should weigh more than 10kb.
To solve this problem, I decided to use a declarative way to describe the components. This has a positive effect on the final size of the application and allows you to use the tree-shaking directly in the component template.
Playground

2. Truly reactive
I really wanted to implement reactivity as it was done in VueJS. If you change the entity of something, then the data on the page also changes, and if you change the data, the entity changes.
Playground

3. Pure reusable components
Any framework should be able to create life-cycle-aware components. This removes duplication of code and allows the component to be reused in several places.
Playground

4. Conditional render and list render directives
I really like the ReactJS, very straight, the first acquaintance, it was just wonderful. However, if you need nested conditional rendering, you will have to either use component wrappers or make ternary expressions that are difficult to read.
Playground

Benchmark application

Playground

Another frameworks

React (Stack) — https://claudiopro.github.io/react-fiber-vs-stack-demo/stack.html
React (Fiber) — https://radi.js.org/fiber/Fiber%20Example.htm
Stencil.js — https://stencil-fiber-demo.firebaseapp.com/perf.html

Performance test

https://pxyup.github.io/Revact/ - try with throttling x6 times.
https://rawgit.com/krausest/js-framework-benchmark/master/webdriver-ts-results/table.html (old name faster-dom)

Link

https://github.com/PxyUp/Revact - source
https://pxyup.github.io/Revact/ - DEMO
https://github.com/RyuuGan/html2FastDom - Html transpiller

Lightweight replacement of React + MobX

Iurii Panarin — Mon, 25 Mar 2019 18:47:23 +0000

FastDom

Lightweight replacement of React + MobX (I hope in future Angular/Vue), which does not use the virtual DOM comparison, but the re-render of only the changed parts. Abandon the HTML template in favor of their interpretation in JS, give to us tree-shaking is components/templates and the speed of work increases since the time to parse the template is zero.

DEMO LINK
DEMO at StackBlitz

Features

Size - 4.1 kB or 1.38 kB gzipped.
The library rewrites only changes and only when it is necessary.
Performance - guarantee 60 fps.
Names of imported functions and classes are not finally and can be discussed.
There is a tree-shaking for components and templates !!!.

Vue conditional render based on breakpoint

Iurii Panarin — Tue, 23 Oct 2018 08:35:31 +0000

Good day all! I was create directive for conditionals renders based on screen size, support resize event.
https://github.com/PxyUp/vue-not-visible
https://www.npmjs.com/package/vue-not-visible
Demo: https://pxyup.github.io/vue-not-visible/index.html

Angular Virtual Table

Iurii Panarin — Mon, 22 Oct 2018 23:54:22 +0000

Good evening, I created virtual scroll table based on material virtual scroll. This table support filtering, sorting and you can easy customize that. Demo included

https://github.com/PxyUp/ng-virtual-table

Demo: https://pxyup.github.io/ng-virtual-table

In future planned so many options like drag and drop, more filter types and more like that. Thanks for any help :)