DEV Community: Finiam

Building dapps with Starknet React

David Lange — Mon, 12 Jun 2023 13:40:00 +0000

Parts of this article were originally written for The Starknet Book. Check it out for more content on Starknet development.

The good news

If you've been following Starknet's progress, you're probably aware that there has been a lot of change recently. Cairo 1 was released, and that means frontend tools need updating to keep up.

If you have experience building dapps but are new to Starknet, the good news is that you can apply plenty of that knowledge here! If you're familiar with Starknet, the good news (in my opinion) is that the changes are moving in the right direction.

Getting started

The basic tools you can use to get started are Starknet.js and get-starknet. Starknet.js is an essential low level lib, equivalent to ethers.js, whereas get-starknet handles connecting to wallets. You should definitely read Starknet.js' docs for a more in-depth understanding.

With these tools, there are basically 3 main concepts to know on the frontend:

Account

We can generally think of the account as the "end user" of a dapp, and some user interaction will be involved to gain access to it.

Think of a dapp where the user connects their browser extension wallet (such as ArgentX or Braavos) - if the user accepts the connection, that gives us access to the account and signer, which can sign transactions and messages.

Unlike Ethereum, where user accounts are Externally Owned Accounts, Starknet accounts are contracts. This might not impact your dapp's frontend, but you should definitely be aware of this difference.

async function connectWallet() {
    const starknet = await connect();
    console.log(starknet.account);
    starknet.account.signMessage(...)
}

The snippet above uses the connect function provided by get-starknet to establish a connection to the user wallet.

Provider

This one is pretty simple. The provider allows you to interact with the Starknet network. You can think of it as a "read" connection to the blockchain, with methods such as getBlock or getChainId.

Just like in Ethereum, you can use a default provider, or use services like Infura or Alchemy, both of which support Starknet.

export const provider = new Provider({
  sequencer: {
    network: "goerli-alpha",
  },
  /* rpc: {
    nodeUrl: INFURA_ENDPOINT
  } */
});

const block = await provider.getBlock("latest"); // <- Get latest block    
console.log(block.block_number);

Contracts

Of course, your frontend will likely be interacting with deployed contracts. For each contract, there should be a counterpart on the frontend. To create these instances, you will need the contract's address and ABI, and either a provider or signer.

const contract = new Contract(
  abi_erc20,
  contractAddress,
  starknet.account
);

contract.balanceOf(starknet.account.address)

If you create a contract instance with a provider, you'll be limited to calling read functions on the contract - only with a signer can you change the state of the blockchain.

Units

If you have previous experience with web3, you know dealing with units can be tricky, and Starknet is no exception. Once again, the docs are very useful here, especially this section on data transformation.

Very often you will need to convert Cairo structs (such as Uint256) that are returned from contracts into numbers:

// Uint256 shape:
// { 
//    type: 'struct', 
//    low: Uint256.low, 
//    high: Uint256.high 
// 
// }
const balance = await contract.balanceOf(address); // <- uint256
const asBN = uint256.uint256ToBN(uint256); // <- uint256 into BN
const asString = asBN.toString() //<- BN into string

And vice versa:

const amount = 1;

const amountFormatted = {
    type: "struct",
    ...uint256.bnToUint256(amount),
};

If we put it all together, calling a transfer function on a contract looks like this:


const tx = await contract.transfer(recipientAddress, amountFormatted);
//or: const tx = await contract.invoke("transfer", [recipientAddress, amountFormatted]); 

console.log(`Tx hash: ${tx.transaction_hash}`);

There are other helpful utils, besides bnToUint256 and uint256ToBN, provided by Starknet.js.

We now have a solid foundation to build a Starknet dapp! If you have a background in regular Ethereum apps, you'll notice how similar the experience is, but you're probably also wondering if there are more high-level tools out there that we can use. Well, there are!

Starknet React

Starknet React is an open-source collection of React providers and hooks for Starknet, inspired by wagmi. If you've used wagmi before, you should have a pretty good idea of what it looks like, as it aims to have a very similar API.

To explore an example project showcasing a dapp built with Starknet React, check out the starknet-demo-dapp repo.

To use Starknet React, start by installing the necessary dependencies:

yarn add @starknet-react/core starknet get-starknet

Starknet.js is an SDK for interacting with Starknet, whereas get-starknet is a package for handling wallet connections.

Then, wrap your app in a StarknetConfig component. This allows for some configuration and provides a React Context for the underlying app to be able to use the shared data and hooks. StarknetConfig receives a connectors prop that defines which wallet connection options will be available to the user.

const connectors = [
  new InjectedConnector({ options: { id: "braavos" } }),
  new InjectedConnector({ options: { id: "argentX" } }),
];

return (
    <StarknetConfig
      connectors={connectors}
      autoConnect
    >
      <App />
    </StarknetConfig>
)

Connection and account

You can now use a hook to access the connectors you defined in the config, allowing the user to connect their wallet:

export default function Connect() {
  const { connect, connectors, disconnect } = useConnectors();

  return (
    <div>
      {connectors.map((connector) => (
        <button
          key={connector.id()}
          onClick={() => connect(connector)}
          disabled={!connector.available()}
        >
          Connect with {connector.id()}
        </button>
      ))}
    </div>
  );
}

Note that there is also a disconnect function provided, in order to end the connection. Once the user has connected their wallet, you have access to the connected account through the useAccount hook. This also provides the current state of connection:

const { address, isConnected, isReconnecting, account } = useAccount();

return (
    <div>
      {isConnected ? (
          <p>Hello, {address}</p>          
      ) : (
        <Connect />
      )}
    </div>
);

Note that state values such as isConnected and isReconnecting are updated automatically, making it easier to update the UI conditionally. This is a very useful and common pattern when dealing with asynchronous processes, as it means you don't have to manually keep track of that state locally in your components.

Once the user is connected, signing messages may be done through the account value returned from the useAccount hook, or more simply through the useSignTypedData hook.

const { data, signTypedData } = useSignTypedData(typedMessage)  

return (
  <>
    <p>
      <button onClick={signTypedData}>Sign</button>
    </p>
    {data && <p>Signed: {JSON.stringify(data)}</p>}
  </>
)

You can sign an array of BigNumberish, or an object. In the event of signing an object, the data must be correctly typed, in accordance with EIP712. You can find a more in depth explanation here.

Network

Starknet React also provides hooks for interacting with the provider. For instance, useBlock fetches the latest block:

const { data, isError, isFetching } = useBlock({
    refetchInterval: 10_000,
    blockIdentifier: "latest",
});

if (isError) {
  return (
    <p>Something went wrong</p>
  )
}

return (
    <p>Current block: {isFetching ? "Loading..." : data?.block_number}<p>
)

The refetchInterval specifies how often the hook will attempt to refetch the data. Under the hood, Starknet React uses react-query for state and query management. Besides useBlock, there are other hooks that can be configured to periodically update, such as useContractRead and useWaitForTransaction.

With the useStarknet hook, it's also possible to directly access the ProviderInterface:

const { library } = useStarknet();

// library.getClassByHash(...)
// library.getTransaction(...)

Interacting with contracts

Read functions

Just like with wagmi, there is a useContractRead hook provided specifically for calling read functions on contracts. Note that this may be used even with without a connected user, as read functions don't require a signer.

const { data: balance, isLoading, isError, isSuccess } = useContractRead({
    abi: abi_erc20,
    address: CONTRACT_ADDRESS,
    functionName: "allowance",
    args: [owner, spender],
    // watch: true <- refresh at every block
});

For working with ERC20s, there is also a convenience hook - useBalance. This hook doesn't require passing in an ABI, and will return a correctly formatted balance value.

  const { data, isLoading } = useBalance({
    address,
    token: CONTRACT_ADDRESS, // <- defaults to the ETH token
    // watch: true <- refresh at every block
  });

  return (
    <p>Balance: {data?.formatted} {data?.symbol}</p>
  )

Write functions

For write functions, the useContractWrite hook is slightly different to wagmi. Due to Starknet's architecture, accounts can natively support multicall transactions. In practice, this means an improved user experience when executing multiple transactions, as you won't have to individually approve each transaction. Starknet React makes the most of this feature through the useContractWrite hook. It can be used in the following manner:

const calls = useMemo(() => {
    // compile the calldata to send
    const calldata = stark.compileCalldata({
      argName: argValue,
    });

    // return a single object for single transaction, 
    // or an array of objects for multicall**
    return {
      contractAddress: CONTRACT_ADDRESS,
      entrypoint: functionName,
      calldata,
    };        
}, [argValue]);


// Returns a function to trigger the transaction
// and state of tx after being sent
const { write, isLoading, data } = useContractWrite({
    calls,
});

function execute() {
  // trigger the transaction
  write();
}

return (
  <button type="button" onClick={execute}>
    Make a transaction
  </button>
)

In the example above, we first compile the calldata to be executed using Starknet.js's compileCalldata util. We pass the contract address, entrypoint, and calldata to the useContractWrite hook, which returns a write function we can use to actually trigger the transaction. The hook also returns the hash and state of the contract call.

Single instance of a contract

Using useContractRead and useContractWrite might not be a good fit for your use case - you might want to work with a single instance of the contract, instead of continually specifying its address and ABI in individual hooks. This is also possible throught the useContract hook:

const { contract } = useContract({
    address: CONTRACT_ADDRESS,
    abi: abi_erc20,
});

// Call functions directly on contract
// contract.transfer(...);
// contract.balanceOf(...);

Transactions

Once you have a transaction hash, you may track it's state through the useTransaction hook. This keeps a cache of all transactions, reducing duplicated network requests.

const { data, isLoading, error } = useTransaction({ hash: txHash });

return (
  <pre>
    {JSON.stringify(data?.calldata)}
  </pre>
)

Wrapping up

You can find a working example with any of these hooks implemented in this repo.

Expect more changes in the next few months, as the Starknet ecosystem continues to evolve!

Happy hacking 👋

Storing custom structs in Cairo 1

Davide Silva — Mon, 29 May 2023 23:00:00 +0000

Let's imagine you are writing a Starknet smart contract in Cairo 1 and you want to store some complex data as the value of a map in your contract's storage. You probably want to use a custom struct to save that complex data but you'll find a small surprise.

Custom Struct

You might define your custom struct like this

#[derive(Drop, Serde)]
struct CustomStruct {
    a: u64,
    b: u64,
    c: u64,
}

and use it on the Storage struct like so

struct Storage {
    map: LegacyMap<felt252, CustomStruct>,
}

So far, so good. This makes perfect sense and it's the correct way of doing it.
However, depending on your Cairo version, and at the time of writing, when you attempt to compile your contract, you might get the following error.

Detailed error information: error: Trait has no implementation in context: core::starknet::storage_access::StorageAccess::<hello_starknet::contracts::hello_starknet::HelloStarknet::CustomStruct>
 --> contract:76:54
            starknet::StorageAccess::<CustomStruct>::read(
                                                     ^**^

error: Trait has no implementation in context: core::starknet::storage_access::StorageAccess::<hello_starknet::contracts::hello_starknet::HelloStarknet::CustomStruct>
 --> contract:84:54
            starknet::StorageAccess::<CustomStruct>::write(
                                                     ^***^

What's the problem here? What this means is simply the StorageAccess Trait has no implementation of the write and read functions needed to access your CustomStruct in the Storage struct. For native types, this is automatically derived but since you defined your own CustomStruct, the compiler has no idea what to do.

Note: this is a temporary problem and in newer versions of Cairo we no longer need to manually derive the implementation. This PR adds that functionality for custom structs used in Storage and is already available if you are using the main branch of the Cairo compiler.

Implementation

So, while we wait for a proper release to include this functionality, how can we fix this? Let's take a look at a possible implementation.

We start by defining a new implementation of the StorageAccess for our CustomStruct.

impl CustomStructStorageAccess of StorageAccess::<CustomStruct> {}

Inside, we define two functions, a write function and a read function. As you can see from the signatures, the write funtion receives, amongst other parameters, a CustomStruct that we want to save in our Storage space, and the read function returns a stored CustomStruct wrapped on a SyscallResult.

fn write(address_domain: u32, base: StorageBaseAddress, value: CustomStruct) -> SyscallResult::<()> {}
fn read(address_domain: u32, base: StorageBaseAddress) -> SyscallResult::<CustomStruct> {}

Write

A write function implementation could look something like this. We receive a CustomStruct and we make use of the storage_write_syscall function to write each individual value to our storage space.

fn write(address_domain: u32, base: StorageBaseAddress, value: CustomStruct) -> SyscallResult::<()> {
  storage_write_syscall(
    address_domain,
    storage_address_from_base_and_offset(base, 0_u8),
    value.value_a.into()
  );

  storage_write_syscall(
    address_domain,
    storage_address_from_base_and_offset(base, 1_u8),
    value.value_b.into()
  );

  storage_write_syscall(
    address_domain,
    storage_address_from_base_and_offset(base, 2_u8),
    value.value_c.into()
  )
}

One thing to take into consideration is the use of the storage_address_from_base_and_offset function. This function is responsible for generating the address of our storage space where we are saving our element. Since our struct has three elements, the second parameter of the function goes from 0_u8 to 2_u8, so that we have a unique spot for each element.

Read

The read function is what we need in order to retrieve the stored data of our CustomStruct. This function makes use of the storage_read_syscall function to retrieve each of our elements from the storage space. We also need to provide the same offset as we did for the write function and we simply construct a CustomStruct instance with the data retrieved. We wrap our struct in a Result::Ok due to the function signature expected from the StorageAccess trait.

One possible implementation would be something like this.

fn read(address_domain: u32, base: StorageBaseAddress) -> SyscallResult::<CustomStruct> {
  Result::Ok(
    CustomStruct {
      value_a: storage_read_syscall(
        address_domain,
        storage_address_from_base_and_offset(base, 0_u8)
      )?.try_into().expect('not u64'),
      value_b: storage_read_syscall(
        address_domain,
        storage_address_from_base_and_offset(base, 1_u8)
      )?.try_into().expect('not u64'),
      value_c: storage_read_syscall(
        address_domain,
        storage_address_from_base_and_offset(base, 2_u8)
      )?.try_into().expect('not u64'),
    }
  )
}

Wrapping up

After adding this implementation, our contract should compile without any problem. For reference, here is the full implementation of a contract that uses a custom struct using this manual implementation approach.

Once again, this is only needed at the time of writing, and if we are using tagged release versions. On the main branch, we no longer need to provide the impl block to use this functionality, simply defining the struct is enough.

That's pretty much it. This was just a short post to show how we can manually implement the read and write function for custom defined types.

Until next time!

Sign in with Metamask using LiveView

José Diogo Viana — Fri, 05 May 2023 09:38:24 +0000

There's a chance that you have heard of Metamask in the past year or two. But, if you haven't, don't worry, as I'll try to explain it quickly. Metamask is an Ethereum crypto wallet, and serves as a gateway to dApps (decentralized Apps). Coinbase has the following definition for a crypto wallet:

Crypto wallets store your private keys, keeping your crypto safe and accessible. They also allow you to send, receive, and spend cryptocurrencies like Bitcoin and Ethereum.

For a more complete explanation, I recommend reading their full blog post, but for the ones that won't read it, I want to highlight that Metamask is just one option between several existing Hot Wallets (Online Wallets). However, for various reasons that don't matter here, Metamask grew into one of the most used wallets.

The main idea of my discussion today is not limited to the Metamask wallet only but can be applied to any other wallet as well. However, you'd need to adjust the code a bit for other wallets, as they may handle Javascript events differently. If you want to support multiple wallets in your dApp instead of doing all the manual work for each one individually, it's probably best to look into things like RainbowKit, wagmi, etc.

My teammate David Lange did an awesome talk for Coimbra.Blockchain where he talks about some of the hurdles and tools to use when developing a frontend for a dApp. I definitely recommend watching the presentation video. (It takes about 40 minutes.)

Why is it relevant?

By using a solution such as Sign in with Metamask (or any other wallet, as highlighted before) the user doesn't need to worry about an extra username + password combo for our App. Yes, I know that there are password managers, and it makes these worries a bit redundant, but let's be honest... End users of these apps don't want to trust us with their emails and passwords, even if they are random. Also, they already need to have a wallet to interact with other Blockchain features, so we might as well take advantage of that.

This can be used as a reliable authentication method because we know that only the person with access to that wallet's private key will be able to cryptographically sign a message, that when verified it'll correspond to the public key. Side note: in Ethereum, and some other blockchains, the public key corresponds to the Wallet's Address. We'll see below how this works with a bit more detail.

How does it work?

If you are still here, then I'm going to assume that you're familiar with how Asymmetric Encryption works (read this post if not).

The image below tries to represent how a crypto wallet can be used as a valid sign-in mechanism. The application's backend gives the User a message to sign with the wallet's private key. The backend then receives the encrypted message and uses the Wallet's Public Key (its Address, for Ethereum) to decrypt it and verify it against the original one.

Some of the more trained eyes may have seen a possible issue with the scheme above. Bear with me for a little more, because I'm going to address it in some paragraphs below.

An important note for this blog post is that the flow I'm talking about is for situations where we have a dedicated backend, and we want to maintain some kind of session between our App and the User. It's very frequent to have dApps that don't need a dedicated backend, so the User just connects its Wallet to the frontend, and can then interact directly with Smart Contracts running in the blockchain.

In the same way, Metamask gives you an interface to make transactions in the blockchain, it also allows the User to just sign simple text messages, using their private keys. The following image is one example. This action is completely off-chain, so therefore it doesn't have associated costs or delays.

Why Elixir and Liveview?

Well, why not? That's a valid reason, but I like to think that I have a more elaborate answer for this. To start, most projects in the Web3/Crypto space have separate frontends from the backend (at least the ones that have a dedicated backend). The reason for this is mainly because there are a lot of tools that can help with quick development if using any modern Javascript Framework.

In my case, I just wanted to do a simple API Playground where any user could interact with the project I was working on, but where it'd be possible to have an authenticated place. The API was all built with Elixir, and Phoenix 1.7 was a couple of days from being released, so I decided to take the opportunity and do something using LiveView, and have a learning experience.

A look into the desired Flow

I can't say I have used many crypto Wallets, but most of them work pretty much the same way. Before interacting with a dApp, you first need to connect the wallet to that website. I'll explain better what this is, down the line, but it's a required step regardless if you want the app to trigger any action from the wallet.

Focusing only on our goal of implementing a Sign In with Metamask flow, the base flow should look like this:

Connect the wallet;
Sign a message;
Validate the signature on the backend;
Create a Session token, if the signature is valid.

Above, we have already seen how we can sign messages using Metamask, but there is a problem with the flow I described... In that example, the user signs a message saying Here is a message!, and then the backend validates the signature, checking if it matches the original message. Even if you don't know much about cryptography, you can understand that if you always sign the same message contents, the resulting signature is the same, also. So, if your key is A and you are signing B, the final signature is always C. This way, if an attacker can intercept the Signature going from Metamask to the backend, then it'd be able to impersonate the actual wallet owner and sign in with our Application because the signature would be valid!

To solve that problem, we need to have a Nonce associated with each User in the database. With it, we make sure that every time a user is signing a message to log in to our app, that message has an extra number/string to make it different every time, therefore resulting in different signatures. Then, as our backend knows the base message, and the nonce given to the respective user trying to sign in, it's still possible to fully validate if the signature is valid or not.

All good up until now? It's time to dive into some code and see how all of this glues together.

Getting our hands dirty

The first step I took was to look for existing examples using Elixir and LiveView. I end up finding this interesting presentation, by Crystal Adkins, talking about what I wanted to learn about! The presentation links to the respective Github Repo of the project demonstrated during it. After exploring for a bit, I was able to understand how the interactions between LiveView and Metamask worked through JS, so I started working on my own version of the code.

Note: There are some differences between the project presented by Crystal Adkins and our use case. For example, you first had to sign in/up using email and password, and only then you could associate and connect Metamask with your account. In that codebase, you always sign the same message (the problem I mentioned), but I assume that it was just a Proof of Concept of the capabilities of LiveView. The same project also did more complex stuff like actually interacting with Smart Contracts, something that I'm not talking about today, and it wouldn't be present in our situation.

Second Note: The message structure being signed in my version, doesn't follow any Ethereum Improvement Proposals (EIPs) like EIP-4361. EIPs describe standards for the Ethereum platform, including core protocol specifications, client APIs, and contract standards. To implement something like that, you'd just need to change the message contents and add more information for User signing it.

To manage sessions and all the corresponding logic, I decided to use the mix phx.gen.auth Accounts User users generator, and remove all of the unwanted parts (like account confirmation, reset password flows, etc...). It's a pretty standard approach and works just fine.

Then I started the actual implementation of the Sign In button. This button is a LiveView component, and has the following mount function:

defmodule DoraWeb.MetamaskButtonLive do
  use DoraWeb, :live_view

  @impl true
  def mount(_params, _session, socket) do
    {:ok,
     assign(socket,
       connected: false,
       current_wallet_address: nil,
       signature: nil,
       verify_signature: false
     )}
  end

  # render logic and event handlers...
end

Looking at the assigns and explaining what each of them represents, is very simple:

connected: if the Metamask Wallet is already connected to our website or not. (The connect thing, I mentioned before). We can get this information from window.ethereum, and I'll explain what it is, next.
current_wallet_address: if the Wallet is already connected, or the User just connected, we can retrieve which Address it corresponds to. (Don't forget that here the Address is also the Public Key.)
signature: after triggering a signing request, this assign will hold the signature to be sent to the backend, for verification.
verify-signature: is supposed to work as a flag to trigger the verification process or not.

The render function is also understandable at first look. If we have @connected as true, then we can render the Sign In message, trigger a new Signature Request, and then the respective verification. If @connected is false, then we render Connect, and clicking on the button triggers the Wallet connection, instead.

@impl true
def render(assigns) do
  ~H"""
  <span title="Metamask" id="metamask-button" phx-hook="Metamask">
    <%= if @connected do %>
      <.form
        for={%{}}
        action={~p"/users/log_in"}
        as={:user}
        phx-submit="verify-current-wallet"
        phx-trigger-action={@verify_signature}
      >
        <.input type="hidden" name="public_address" value={@current_wallet_address} />
        <.input type="hidden" name="signature" value={@signature} />
        <.button class={button_css()}>
          <span class="w-6"><.metamask_icon /></span> Sign in
        </.button>
      </.form>
    <% else %>
      <.button class={button_css()} phx-click="connect-metamask">
        <span class="w-6"><.metamask_icon /></span> Connect
      </.button>
    <% end %>
  </span>
  """
end

Let's start by inspecting phx-click="connect-metamask" for the case where the Wallet is not connected. It relies on the push_event function. It's a small function, but very powerful in that way that enables us to easily communicate with the Javascript side of the App, by pushing an event to the client, and where we can interact directly with Metamask.

@impl true
def handle_event("connect-metamask", _params, socket) do
  {:noreply, push_event(socket, "connect-metamask", %{})}
end

How to interact with Metamask from Javascript?

MetaMask injects a global API into websites visited by its users at window.ethereum. This API allows websites to request users' Ethereum accounts, read data from blockchains the user is connected to, and suggest that the user sign messages and transactions.

Consequently, to interact with window.ethereum, we have Javascript libraries like Ethers or web3.js. I end up using Ethers, but you can select whatever you see fit. To handle the event above, you need something like:

const web3Provider = new ethers.providers.Web3Provider(window.ethereum)

window.addEventListener(`phx:connect-metamask`, (e) => {
    web3Provider.provider.request({method: 'eth_requestAccounts'}).then((accounts) => {
      if (accounts.length > 0) {
        signer.getAddress().then((address) => {
            this.pushEvent("wallet-connected", {public_address: address})
        });
      }
    }, (error) => console.log(error))
})

Upon successful connection of the Wallet, a new event "wallet-connected" is pushed with the respectively connected address. We can handle this event in our LiveView file, just like any other emitted by our template. When receiving that event, we set @connected to its new state, and we also update @current_wallet_address. By changing these two assigns the button will now render a different thing entirely because the Wallet is already connected to our app.

@impl true
def handle_event("wallet-connected", params, socket) do
  {:noreply,
   assign(socket,
     connected: not is_nil(params["public_address"]),
     current_wallet_address: params["public_address"]
   )}
end

Since we are already connected, when clicking our new button, the event emitted when trying to submit the form is phx-submit="verify-current-wallet". Even though this form is a post to ~p"/users/log_in", this action will only be triggered when @verify_signature is true, due to phx-trigger-action={@verify_signature} (check the px-trigger-action docs for more info on how it works).

The handler for this event looks for an existing account to return its current nonce, or if an account for that Address doesn't exist, then it generates a new nonce in the spot. This way, we don't distinguish if a User trying to sign a message to log in will have a valid account or not (the process will fail later instead). This nonce is then pushed to the Javascript client side and will be used for the signing process.

@impl true
def handle_event("verify-current-wallet", _params, socket) do
  nonce =
    case Accounts.get_user_by_eth_address(socket.assigns.current_wallet_address) do
      nil -> Accounts.generate_account_nonce()
      user -> user.nonce
    end

  {:noreply, push_event(socket, "get-current-wallet", %{nonce: nonce})}
end

The Javascript code is actually very similar, and through signer.signMessage(message) a window, similar to the image shown in the introduction part, will appear and the User will be able to sign that message. If the user ends up signing the message, we can push a new event with the result in the payload, and listen to it on the LiveView component side.

As a separate note, there are several events triggered by Metamask (and not our app) that you can also listen to, and make everything feel even snappier. I didn't dive into details, but here you can find docs for API description and the full list of events.

window.addEventListener(`phx:get-current-wallet`, (e) => {
    signer.getAddress().then((address) => {
        const message = `You are signing this message to sign in with Dora. Nonce: ${e.detail.nonce}`

        signer.signMessage(message).then((signature) => {
            this.pushEvent("verify-signature", {public_address: address, signature: signature})

            return;
        })
    })
})

Then, the event pushed by Javascript is handled again on the LiveView Component, and @verify_signature is marked as true. This way, the form submission is triggered at this time, and the post to ~p"/users/log_in" is done.

@impl true
def handle_event("verify-signature", params, socket) do
  {:noreply,
   assign(socket,
     signature: params["signature"],
     verify_signature: true
   )}
end

On the Session controller's side, we receive the form params (public_address and signature), and we verify the signature. If it's valid we call UserAuth.log_in_user(user, params), if it's not, we redirect back to ~p"/".

To verify the User's signature, ExWeb3EcRecover is being used. This is a "simple" library that exports one function recover_personal_signature/2. This function returns the address that created the signature for a personally signed message by an Ethereum Wallet. So, in our case, we can reconstruct the original messages by the User (we have its nonce stored in the DB still), and then we check if the Address trying to sign in with that signature is the same that signed the message in the first place. If so, we update the nonce to a new random value, and we return the user model as success.

def verify_message_signature(eth_address, signature) do
  with user = %User{} <- find_user_by_public_address(eth_address) do
    message = "You are signing this message to sign in with Dora. Nonce: #{user.nonce}"

    signing_address = ExWeb3EcRecover.recover_personal_signature(message, signature)

    if String.downcase(signing_address) == String.downcase(eth_address) do
      update_user_nonce(eth_address)
      user
    end
  end
end

ExWeb3EcRecover depends on ex_keccak, which means you'll need to have Rust installed to compile it. For a quick and easy Rust install follow these instructions.

As you probably have noticed, in our flow no new accounts are created. That's because, for this project, only pre-configured wallets should be able to sign in. However, the change needed to have that working as a sign up should be pretty simple. For example, in the code snippet for the wallet connected, we can do something like this:

@impl true
def handle_event("wallet-connected", params, socket) do
  address = params["public_address"]
  # also validate if is not nil! The User may not connect the wallet
  Accounts.create_user_if_not_exists(address) 

  {:noreply,
   assign(socket,
     connected: not is_nil(address),
     current_wallet_address: address
   )}
end

Wrapping up

This post had a bunch of code that may be hard to follow, but I tried to explain it as best as I can. Now, to fill in the gaps, feel free to look at the full code where this was implemented and the final result:

Github Repo
Example App deployed with Fly.io.

If you have any questions or comments, reach out in one of my socials. And as always, stick around for the next one 👋

A look into Cairo 1.0 traits

Davide Silva — Fri, 28 Apr 2023 10:56:30 +0000

Starknet Regenesis is currently underway. There's been an effort from Starkware into improving not only the Cairo language but the entire ecosystem. We are currently in the transition period but there is already some new language features that we can play with. Let's look at some of the new features of Cairo 1.

Cairo likes Rust

If you have been following the updates from the Starkware team, you probably already know that the new Cairo 1 syntax is going to be very similar to Rust. Not only in terms of syntax but also some of the concepts introduced by Rust are being ported to Cairo, with Traits now being present in Cairo 1.

Traits

What are traits? Traits are a concept in Rust that allow sharing a certain behaviour across types. It's very similar to interfaces in traditional OO languages, but with some key differences. In Cairo, traits don't behave exactly like you would expect if you are familiar with Rust, but it will feel similar.

In Cairo, we can define a trait by using the trait keyword followed by the name of the trait. In the trait block, we specify the signature of the functions that we want that specific trait to implement. Let's look at an example:

trait SuperComputer {
    fn the_ultimate_question() -> felt;
}

We defined our Trait, SuperComputer, and we said that every type that uses this trait needs to implement the the_ultimate_question function.
If a type wants to implement our trait, it will need to implement all the functions declared by our trait definition. Otherwise, the compiler will throw out an error saying Not all trait items are implemented.

A Trait implementation is defined by the impl keyword followed by the name of our implementation and the trait that we are implementing for:

impl DeepThought of SuperComputer {
    fn the_ultimate_question() -> felt {
        42
    }
}

If we now define a new implementation, Marvin, that also uses the SuperComputer trait, we are enforced by the compiler to implement the function the_ultimate_question.

impl Marvin of SuperComputer {
    fn the_ultimate_question() -> felt {
        5
    }
}

Using Generics

The previous example was simply to introduce the concept of Traits. We can also make use of generics in order to make our Trait accept any type we want. We just need to implement our Trait for every type that uses it.

Let's imagine we want to represent a color in our Cairo program. We can follow the widely used RGB color model to define our color. We can have a struct that stores the value of red, green, and blue that make up our color. However, there are also other color models, like the CMYK color model, that use different attributes to represent the same color.

struct RGB {
    red: u8,
    green: u8,
    blue: u8,
}

struct CMYK {
    cyan: u8,
    magenta: u8,
    yellow: u8,
    black: u8,
}

If we wanted to have a way to find the color representation of the color red, using either color model, we could have a ColorModelTrait<T> that would accept any type T, in order to enforce a function red that would return, either a RGB or CMYK struct, with that representation. Like so:

trait ColorModelTrait<T> {
    fn red() -> T
}

impl RGBImpl of ColorModelTrait::<RGB> {
    fn red() -> RGB {
        RGB { red: 255, green: 0, blue: 0 }
    }
}

impl CMYKImpl of ColorModelTrait::<CMYK> {
    fn red() -> CMYK {
        CMYK { cyan: 0, magenta: 100, yellow: 100, black: 0 }
    }
}

With this in place, we can now use our ColorModelTrait to get a struct that will represent the color red. The T in the trait definition represents any possible type so we can specify if we want the RGB or CMYK type when calling our trait.

let rgb = ColorModelTrait::RGB::red();
let cmyk = ColorModelTrait::CMYK::red();

Final thoughts

This brief post aims to show how we can use traits in Cairo. Obviously, there are many more concepts and new syntax to delve into. We are on a learning journey and are excited to share and exchange knowledge with you.

We are currently building a community for anyone interested in Cairo and Starknet in particular. Whether you're an experienced developer or just starting out, join our Starknet Portugal on Telegram and Starknet Portugal on Meetup groups. Don't forget to also follow our socials where we will share more events soon.

Creating animated chickens with zdog

David Lange — Tue, 18 Apr 2023 14:34:23 +0000

Demo: https://rotating-chicken.vercel.app/
Code : https://codesandbox.io/s/rotating-chicken-pcj9w1

Some time ago I found zdog - it's a lightweight 3D javascript engine, for canvas or SVG. If you check out its site, you'll see it described as pseudo 3D. What does that mean? Basically zdog holds a model of 3D shapes, but renders everything as 2D flat shapes.

What it is not

You should know upfront that zdog has its limitations. There's no collision detection or physics, no gradients, no perspective or cameras, no lighting, and a bunch of other stuff you might want. If that's what you need, you might want to look at more powerful and real-3D engines, such as three-js.

What it is

Sounds pretty limited right? However, within these constraints, you can actually do some pretty amazing stuff with zdog, without all the overhead and complexity inherent to three-js. Just take a look at the demos page - it's got a bunch of neat examples of what zdog is best at: a flat, rounded aesthetic, with simple shapes. This makes it very designer-friendly.

It also means that it's very easy to make things actually look good. No more messing around trying to figure out if you should use MeshLambertMaterial or MeshPhongMaterial, or where to place your spot lights or area lights in relation to the scene. Just create a shape and give it a color.

Also, round shapes are actual circles - this means none of the polygon jaggedness you get in real 3D.

Ease of use

You'll find zdog is really easy to use. It has a declarative API that lets you describe the shapes you want, and add them to the scene. The example in the docs describes this well.

First, you add a canvas or svg:

<canvas class="zdog-canvas" width="240" height="240"></canvas>

Then create the shapes and add them to the scene:

// create illustration
let illo = new Zdog.Illustration({
  // set canvas with selector
  element: '.zdog-canvas',
});

// add circle
new Zdog.Ellipse({
  addTo: illo,
  diameter: 80,
  stroke: 20,
  color: '#636',
});

// update & render
illo.updateRenderGraph();

The chicken

So let's get to the chicken. I won't include all the code (you can find that in the sandbox), but I'll include the main parts. First, we need to figure out what individual shapes we need. In this case, it looks something like this:

Knowing this, we can start adding shapes:

export let model = new Zdog.Illustration({
  element: ".zdog-canvas",
  resize: true,
  rotate: { x: rotation.initialX, y: rotation.initialY },
  onResize: function (width, height) {
    this.zoom = Math.floor(Math.min(width, height) / sceneSize);
  },
});

let bodyGroup = new Zdog.Group({
  addTo:  model
});

// head
new Zdog.Hemisphere({
  stroke: 6,
  diameter: 15,
  addTo: bodyGroup,
  color: white,
  rotate: { x: Zdog.TAU / 4 },
});

// neck
new Zdog.Cylinder({
  stroke: 6,
  diameter: 15,
  length: 14,
  addTo: bodyGroup,
  color: white,
  frontFace: white,
  backface: white,
  rotate: { x: Zdog.TAU / 4, z: Zdog.TAU },
  translate: { y: 8 },
});

// beak
new Zdog.Cone({
  addTo: model,
  diameter: 4,
  length: 5.5,
  stroke: 2,
  color: yellow,
  backface: yellow,
  rotate: { y: -Zdog.TAU / 4 },
  translate: { x: 11, y: 4 },
});

// beard thing
new Zdog.Shape({
  addTo: model,
  stroke: 3.2,
  path: [
    { x: 11, y: 9 },
    { x: 11, y: 11 },
  ],
  color: red,
});

As you can see, creating shapes is easy. Besides size and color, we can set the translation and rotation. We can add shapes directly to the main model object, or add them to shapes that are then added to model.

Next, we can add the eyes:

// eyes group
let eyesGroup = new Zdog.Group({
  addTo: model,
});

// left eye
let eye1 = new Zdog.Shape({
  addTo: eyesGroup,
  stroke: 1.5,
  color: darkGrey,
  translate: { x: 8, y: -1, z: 3 },
});

// right eye
eye1.copy({
  translate: { x: 8, y: -1, z: -3 },
});

And finally the crest:

// path for each shape
const crestData = [
  {
    path: [
      { x: -4.5, y: -11 },
      { x: -5, y: -11.5 },
    ],
  },
  ...
];


let crestGroup = new Zdog.Group({
  addTo: model,
});

// create all crest shapes
crestData.forEach((item) => {
  new Zdog.Shape({
    addTo: crestGroup,
    stroke: 4.5,
    color: red,
    ...item,
  });
});

At this point, we should have a decent looking static chicken!

Animating things

Next, we can add animation. In this scene, there are 2 animations: blinking and jumping.

As per the docs, this can be done by adding a "render loop" function, that applies changes to the scene. This function runs on every frame, via requestAnimationFrame:

function animate() {
  // rotate model each frame
  model.rotate.y += 0.03;
  model.updateRenderGraph();
  // animate next frame
  requestAnimationFrame( animate );
}
// start animation
animate();

We can use the same principles for blinking.

But first, how can we visually blink? One method would be to make the eyes invisible and quickly visible again. But a slightly better way is to have a white shape just above the eyes, which we move down (covering the eyes) and then back up (revealing the eyes). Like this we get the eyelid motion, and as the shape is the same color as the chicken's head, we only see it when it's covering the eyes:

// eyelid
export let eyelid = new Zdog.Shape({
  addTo: eyesGroup,
  path: [
    { x: 8, y: -3, z: -3.4 },
    { x: 8, y: -3, z: 3.4 },
  ],
  stroke: 2,
  color: white,
});

To animate shapes, and keeping in line with zdog, I decided to use class ZAnimation:

export class ZAnimation {
  duration: number;
  frame = 0;
  force: number;
  running = false;
  onEnd?: () => void;
  apply: (val: number) => void;

  constructor(options: {
    duration: number;
    force: number;
    interval?: number;
    apply: (val: number) => void;
    onEnd?: () => void;
  }) {
    this.duration = options.duration;
    this.force = options.force;
    this.apply = options.apply;
    this.onEnd = options.onEnd;

    if (options.interval) {
      this.schedule(options.interval);
    }
  }

  schedule(interval: number) {
    setInterval(() => {
      this.running = true;
    }, interval);
  }

  start() {
    this.running = true;
  }

  stop() {
    this.running = false;
  }

  handleFrame() {
    if (!this.running) return;

    this.frame++;

    this.apply(this.force);

    if (this.frame === this.duration) {
      this.frame = 0;
      this.stop();
      this.onEnd?.();
    }
  }
}

Let's look at the class' properties:

duration: how many frames it should last;
frame: this keeps track of the animation's progress;
force: the amount that the animated shape should change in each frame;
apply: the actual transformation that will happen (such as model.rotate.y += x);
running: whether the animation is running or not. When frame equals duration, this will get set to false, as that is the animation's end;
interval: in case we want the animation to happen periodically (useful when blinking);
onEnd: optional callback that runs when the animation ends. Useful when chaining animations.

We can then use it like this:

const blinkUp = new ZAnimation({
  duration: 18,
  force: 0.1,
  apply: (val) => {
    eyelid.translate.y -= val;
  }
});

const blinkDown = new ZAnimation({
  duration: 18,
  force: 0.1,
  interval: 4000,
  apply: (val) => {
    eyelid.translate.y += val;
  },
  onEnd: () => {
    blinkUp.start();
  }
});

function animate() {
  model.updateRenderGraph();

  blinkDown.handleFrame();
  blinkUp.handleFrame();

  requestAnimationFrame(animate);
}

animate();

We create the animations, and specify that blinkDown should happen every 4 seconds, and when it ends it should call blinkUp. This is what it looks like:

What about jumping? For that we can add a couple more chained animations, and add an event listener so our chicken can jump on click:

const bounceDown = new ZAnimation({
  duration: 28,
  force: 1,
  apply: (val) => {
    model.translate.y += val;
  }
});

const bounceUp = new ZAnimation({
  duration: 28,
  force: 1,
  onEnd: () => bounceDown.start(),
  apply: (val) => {
    model.translate.y -= val;
  }
});

document.addEventListener("click", () => {
  bounceUp.start();
});

Voilá, it jumps.

Easing

Nice! We now have a blinking and jumping chicken. It's good, but you may have noticed that the jumps are kind of...bland. They look too mechanical, because the movement is linear. To make it look natural, we need to add some easing. I recommend you check out easings.net, it's an excellent resource on this subject that I used here. Besides showing what different easing functions look like, it has implementations in CSS and math functions. I actually use those exact functions in this demo.

Basically we want the force applied in each frame to change, depending on what the current frame is relative to the total duration we specified:

In the first graph, the change is linear, meaning it happens in a consistent manner over time (like our chicken right now). On the right we have an "eased" movement, where x will initially increase slowly, then more quickly in the middle of the animation, before slowing down towards the end.

We can add easing to our ZAnimation class:

class ZAnimation {
    // ...
    easing: Easing = "easeOutCirc";
    easeAcc = 0;

    constructor(options: {
        // ...
        easing: Easing;        
    }) {
        // ...
        this.easing = options.easing;        
    }

    // ...

    handleFrame() {
        if(!this.running) return;

        this.frame++;

        const value = this.calcEasedValue();
        this.apply(value);

        if (this.frame === this.duration) {
          this.frame = 0;
          this.easeAcc = 0;
          this.stop();
          this.onEnd?.();
        }
    }

    calcEasedValue() {
        const progressDecimal = (this.frame * 100) / this.duration / 100;

        let easedVal = 0;

        switch (this.easing) {
          case "easeOutCirc":
            easedVal = easeOutCirc(progressDecimal);
            break;

          case "easeOutBounce":
            easedVal = easeOutBounce(progressDecimal);
            break;
        }

        const frameEase = easedVal - this.easeAcc;
        this.easeAcc += frameEase;

        return frameEase * this.force;
    }
}

Let's break this down: when we create the animation we can specify the easing function we want (in this case there are only two options, easeOutCirc and easeOutBounce).

Then, in the handleFrame function, instead of simply using the force property, we calculate the eased force for each specific frame. For this we need to work out the current progress, from 0 to 1:

 const progressDecimal = (this.frame * 100) / this.duration / 100;

and pass that value to the easeOutCirc or easeOutBounce functions, which will in turn return a value between 0 and 1. Where are these functions? Well, I copied them from easings.net! They look something like this:

function easeInOutCubic(x: number): number {
    return x < 0.5 ? 4 * x * x * x : 1 - Math.pow(-2 * x + 2, 3) / 2;
}

Note that the value we get from the easing functions is cumulative, meaning that its output will keep increasing from 0 until it's 1 by the last time we call it, and that's not what we want. We want the incremental change for every frame. In order to do this we keep track of the accumulated amount, and subtract it from what the easing function gives us:

const frameEase = easedVal - this.easeAcc;
this.easeAcc += frameEase;

return frameEase * this.force;

Lastly, we multiply the frame's easing value by force. That's the amount by which our shape will move in each frame.

To use this updated class, we just need to add the easing type to the instances:

const bounceDown = new ZAnimation({
  duration: 32,
  force: 6.8,
  easing: "easeOutBounce",
  addTo: scene,
  apply: (val) => {
    model.translate.y += val;
  },
});

const bounceUp = new ZAnimation({
  duration: 28,
  force: 6.8,
  easing: "easeOutCirc",
  addTo: scene,
  onEnd: () => bounceDown.start(),
  apply: (val) => {
    model.translate.y -= val;
  },
});

The jumps look a lot better now.

Wrapping up

You may have noticed that in the demo the chicken also follows the mouse around. You can take a look at the code needed for that, though I'll not delve into that in this post. Who knows, we'll be back for a part II.

If you're wondering when you'll ever need a pseudo 3D chicken that jumps and blinks, I can't answer that.

But if what you need is a bit of visual flair, and don't require more advanced features, zdog can be an excellent choice. It's very easy to use and provides a set of default shapes and tools that help you get up and running very quickly. Combined with some custom animations and user interactions, you can create a delightful result.

As always, thanks for reading!

Exponential Backoff with Elixir

Resende — Tue, 11 Apr 2023 11:05:11 +0000

When we rely on external services, they may not always be available, may take a long time to respond, or may not be able to fulfil our requests. This can be frustrating, but if it's a one-time request and we only need to query one host, we can simply keep trying until we get a response.

However, if we have multiple providers for the same service and need to query them repeatedly, such as in a scenario where we're tracking prices for different franchises of a store and maintaining a leaderboard of the best prices, we can't rely on the same strategy of continually polling a single service. In these cases, we need a more sophisticated approach, such as exponential backoff, to manage the requests and handle any failures more efficiently.

In some cases, the stores that we're polling may not have the specific item we're looking for, or they may have an unstable server or poor internet connection. This can make it challenging to avoid wasting time on unsuccessful requests.

So how can we prevent permanent polling, active wait, blocking code, and scenarios like this:

you - are you ready yet ?

service - NO.

you - What about now?

service - ..... for the last time NO.

you - And now?

service - still NO.

you - Hey service could you give me ...?

service - Iup here you go!

Getting started

Today I'm gonna talk about exponential back-off, and how to deal with this kind of problem in a very simple way.

Better yet I'm gonna show you some examples in Elixir that make this whole process a lot simpler with OTP.

Exponential backoff is like taking a deep breath and counting to 10 before trying again when you don't succeed at something on the first try.
But instead of counting to 10, you wait a little longer each time you try again, giving yourself and others around you a chance to catch their breath and avoid chaos.

We will use GenServers so the final result ends up in a very simple and elegant implementation.

The premises are:

You make the request:
- The request went through. Great, do the next request in 5 seconds.
- The request didn't go through, we'll try again in 25 seconds (5^2).

Now we could do this to infinity but let's do a smarter implementation with an allowlist and blocklist for the stores that return n unsuccessful requests so we don't have an infinite waiting time for the next request

You make the request :
- The request went through. Great, do the next request in 5 seconds.
- The request didn't go through, we'll try again in 25 seconds (5^2) and you get a ticket (1).
- The request didn't go through and you already have 3 tickets, you're going on the blocklist.

Cool so how do I implement this in Elixir?

Well, it's pretty easy!

We can have this structure :

GenServer for the Stores
GenServer for the Allowlist
GenServer for polling each store

GenServer for the stores

In this, we have the main GenServer that will create the allowlist, and each of the polling store processes.

If you don't know, a GenServer is implemented in two parts: the client API and the server callbacks. If you want to learn more check this blog post on concurrency mechanisms in Elixir that I did a while back.

At the public API level, we only need a way to get the prices. This is the objective of this module, get prices.

defmodule Project.Stores do
 def prices do
    GenServer.call(__MODULE__, :prices)
 end
 # ....
end

On the private callback side, we need to support the public price call (of course) and the refresh callback to update the allow list.

The price call is gonna query the allowlist for the prices of the stores that are currently available and answering to our queries.

defmodule Project.Stores do
  # ...
  # private API
  def handle_call(:prices, _from, %{allowlist: allowlist} = state),
    do: {:reply, Stores.Allowlist.list(allowlist), state}

  def handle_info(:refresh, %{period: period, allowlist:   allowlist} = state) do
    case API.list_stores() do
      {:ok, stores} ->
        start_store_polling(stores, allowlist)

      error ->
        error
    end

    schedule_work(period)

    {:noreply, state}
  end
  # ...
end

Finally, we can have private functions like starting the polling GenServer for each store and a function to periodically wake up the process by sending a message to self.

This message will be caught by the handle_info on the private callback allowing for a refresh of the stores.

defmodule Project.Stores do
  # ...
  defp start_store_polling(stores, allowlist) do
    for store <- stores do
      Stores.Polling.start_link(store, allowlist)
    end
  end

  defp schedule_work(period), do: Process.send_after(self(), :refresh, period)
end

GenServer for the allowlist

This allowlist will both keep all the current stores that are responding as well as their prices and a quarantine list for stores that may be removed if they keep failing to respond. With this we need the public API to support adding a store, adding its price, removing a store, quarantining a store, and finally listing the stores and prices.

defmodule Project.Stores.Allowlist do
  use GenServer

  def start_link,
    do: GenServer.start_link(__MODULE__, :ok)

  def init(:ok),
    do: {:ok, %{allowlist: %{}, quarantine: MapSet.new()}}

  def add(pid, store, price),
    do: GenServer.cast(pid, {:add, store, price})

  def remove(pid, store),
    do: GenServer.cast(pid, {:remove, store})

  def quarantine(pid, store),
    do: GenServer.cast(pid, {:quarantine, store})

  def list(pid),
    do: GenServer.call(pid, :list)
  # ...
end

The private API is self-describing, the cast adds, removes, lists, and changes a store from the allow list to the quarantine list. There's nothing outside of the ordinary here.

defmodule Project.Stores.Allowlist do
  # ...
  # private API 
  def handle_cast({:add, store, price}, state) do
    quarantine = MapSet.delete(state.quarantine, store)
    allowlist = Map.put(state.allowlist, store, price)

    {:noreply, %{state | quarantine: quarantine, allowlist: allowlist}}
  end

  def handle_cast({:remove, store}, state) do
    quarantine = MapSet.delete(state.quarantine, store)
    allowlist = Map.delete(state.allowlist, store)

    {:noreply, %{state | quarantine: quarantine, allowlist: allowlist}}
  end

  def handle_cast({:quarantine, store}, state) do
    allowlist = Map.delete(state.allowlist, store)
    quarantine = MapSet.put(state.quarantine, store)

    {:noreply, %{state | quarantine: quarantine, allowlist: allowlist}}
  end

  def handle_call(:list, _from, state) do
    {:reply, Map.values(state.allowlist), state}
  end
end

GenServer for polling a store

The polling store GenServer itself is where all the fun stuff happens. It keeps track of all the requests and the number of failed attempts to communicate to the store and loads constant values like the number of max_attempts allowed and the timeout/base_timeout.

defmodule Project.Stores.Polling do
  use GenServer

  import Project.Config, only: [config!: 2]

  alias Project.Stores.Allowlist

  def start_link(store, allowlist) do
    name = process_name(store)

    state = %{
      store: store,
      allowlist: allowlist,
      attempts: 0,
      max_attempts: config!(__MODULE__, :max_attempts),
      timeout: 2_000,
      base_timeout: 2_000
    }

    GenServer.start_link(__MODULE__, state, name: name)
  end
  # ...
end

On the init function we pass the state that we received from start_link and call the private function schedule_work that will send a message to this process, and make it do some work (the name is accurate).

defmodule Project.Stores.Polling do
  # ...
  def init(state) do
    schedule_work(state.base_timeout)

    {:ok, state}
  end
  # ...
end

When the process receives a message to do some work it queries the store for the current price. Depending on the result, it's handled differently. This is especially easy with tuples in Elixir using the :ok and :error symbols on the first element.

defmodule Project.Stores.Polling do
  # ...
  def handle_info(:work, state) do
    case query_price(state.store) do
      {:ok, price} ->
        handle_successful_polling(state, price)

      {:error, _reason} ->
        handle_missing_polling(state)
    end
  end
  # ...
end

If the polling is successful we add the store and the price to the allowlist GenServer.
We will reschedule the polling with the base timeout and update the process state to set the timeout to the base timeout with 0 retry attempts.

defmodule Project.Stores.Polling do
  # ...
  defp handle_successful_polling(state, price) do
    Allowlist.add(state.allowlist, state.store, price)
    schedule_work(state.base_timeout)

    new_state = %{state | timeout: state.base_timeout, attempts: 0}

    {:noreply, new_state}
  end
  # ...
end

If the process misses the poll and it's the last attempt we are going to remove it from the Allowlist and update the state to mark the attempts as the max attempts.

defmodule Project.Stores.Polling do
  # ...
  defp handle_missing_polling(
         %{attempts: attempts, max_attempts: max_attempts} = state
       )
       when attempts == max_attempts - 1 do
    Allowlist.remove(state.allowlist, state.store)

    {:stop, :normal, %{state | attempts: max_attempts}}
  end
  # ...
end

If it's not the last attempt and the process misses the polling we quarantine that store, exponentiate the timeout and increase the attempt counter.
After that, we set the new state with the timeout and attempts while scheduling the next poll with the new timeout.

defmodule Project.Stores.Polling do
  # ...
  defp handle_missing_polling(state) do
    Allowlist.quarantine(state.allowlist, state.store)

    timeout = exponential_backoff(state.timeout)
    attempts = state.attempts + 1
    new_state = %{state | timeout: timeout, attempts: attempts}

    schedule_work(timeout)

    {:noreply, new_state}
  end
  # ...
end

At the end of the file, we define the helper functions for this module,
exponential backoff, schedule_work, and process name.

The exponential backoff function is a wrapper around the power function from Erlang with a round of the value, the schedule work function is a send-after to the same process, and the process_name uses the module and store to give a unique name to each process.

defmodule Project.Stores.Polling do
  # ...
  defp exponential_backoff(timeout), do: :math.pow(timeout, 2) |> round()

  defp schedule_work(timeout), do: Process.send_after(self(), :work, timeout)

  defp process_name(store), do: :"#{__MODULE__}-#{store}"
end

Conclusion

This was a fairly technical blog post but I truly hope you learned something useful.
Although the example presented in this post is somewhat specific, with small changes it's possible to adapt it to your specific use case.

Hope this will help you in your next project if an exponential backoff is needed, maybe when you are dependent on some service that rate limits requests with PlugAttack as described in this blog post.

Thank you for reading and if you have any questions or doubts feel free to reach out on Twitter at @Resende_666.

Using custom Svelte stores

David Lange — Tue, 21 Mar 2023 12:12:30 +0000

With Svelte, we get an out of the box solution for state management - stores. According to the docs, a store is any object with a subscribe, unsubscribe, and (optionally) a set method. Svelte provides the functions to easily create readable, writable, or derived stores.

At it's most simple:

const simplePizza = writable({
  ingredients: [],
  ovenTime: 0,
  status: 'idle',
});

export default simplePizza;

Then we could use it like this in a component:

<script lang="ts">
  import simplePizza from "../stores/simplePizza"

  let inputVal: string;

  function handleSubmit() {
    simplePizza.update(st => ({
      ...st,
      ingredients: [...st.ingredients, inputVal]
    }))
  }
</script>

<form on:submit|preventDefault={handleSubmit}>
  <input name="ingredient" bind:value={inputVal} />
  <button type="submit">Add</button>
</form>
<ul>
{#each $simplePizza.ingredients as ingredient}
  <li>{ingredient}</li>
{/each}
</ul>

By using the dollar sign to access the store, we're subscribing to updates in the store's value, and our component will react to any change and show the updated ingredient list (you can check the docs on the dollar sign syntax here, but it basically uses the store's subscribe function, and unsubscribes when appropriate). Outside of components, we can manually subscribe:

const unsubscribe = simplePizza.subscribe((value) => {
    // do something when there are changes
});

unsubscribe()

If you've used Svelte before, you probably already know this. What about custom stores?

Custom stores

In many cases, we're likely to need some logic around the store. For instance, we may want to control how the store is updated, without having that validation in components. This is where custom stores come in handy:

//customPizza.ts
type PizzaStore = {
  ingredients: string[];
  ovenTime: number;
  status: 'idle' | 'preparing' | 'baking' | 'ready';
};

function createPizzaStore() {
  const { set, update, subscribe } = writable<PizzaStore>({
    ingredients: [],
    ovenTime: 0,
    status: "idle"
  });

  function addIngredient(ingredient: string) {
    if (ingredient === 'pineapple') {
      throw new Error('Step away from the pizza');
    }

    update((store) => ({
      ...store,
      ingredients: [...store.ingredients, ingredient],
    }));
  }

  return {
    subscribe,
    addIngredient,
  };
}

const pizzaStore = createPizzaStore();

export default pizzaStore;

Basically, we wrap a store in a function that holds additional logic. By returning the subscribe method, we can subscribe to the store's value in our component, same as before. But we can also return other functions, defining how we want the outside world to interact with our store. You can think of it as adding methods to interact with the underlying writable store - like an API.

As you can see, stores are very flexible, and you can actually combine them with other state management libs. For that, I highly recommend you check out this post on using stores with Zustand, a very nice state management lib.

Wrapping the wrappers

We can actually extend this concept of wrapping a store. Say we have several stores, and we want them all to have some shared functionality, like syncing changes to local storage, or a remote DB. We can create a base store that receives the writable store and wrapper function we wrote previously, and passes some base functions into that wrapper:

// _baseStore.ts
interface WrapperProps<T> extends Writable<T> { 
  _syncToLocal: (key: string) => Unsubscriber;
  _set: (args: Partial<T>) => void;
}

export default function _baseStore<T, I>(
  store: Writable<T>,
  wrapperFn: (args: WrapperProps<T>) => I
): I {
  const { update, subscribe } = store;

  function _set(args: Partial<T>) {
    store.update((st) => ({
      ...st,
      ...args,
    }));
  }

  function _syncToLocal(key: string) {
    return subscribe((val) => {
      localStorage.setItem(key, JSON.stringify(val));
    });
  }

  return wrapperFn({ ...store, _syncToLocal, _set });
}

We then pass it our custom store:

// customPizza.ts
type PizzaStore = {
  ingredients: string[];
  ovenTime: number;
  status: 'idle' | 'preparing' | 'baking' | 'ready';
};

const store = writable<PizzaStore>({
  ingredients: [],
  ovenTime: 0,
  status: "idle"
});

const pizzaStore = _baseStore(
  store,
  ({ _syncToLocal, subscribe, update }) => {

    //use the received function
    _syncToLocal('myPizza');

    function addIngredient(ingredient: string) {
      if (ingredient === 'pineapple') {
        throw new Error('Step away from the pizza');
      }

      update((st) => ({
        ...st,
        ingredients: [...st.ingredients, ingredient],
      }));
    }

    return {
      subscribe,
      addIngredient,
    };
  }
);

export default pizzaStore;

As you can see, the wrapper contains all the same logic as before, except now it also receives _syncToLocal. In this way, we can move that sort of common functionality to another layer, where it can be easily reused by other stores. Let's say you also have cupcakes in your app:

const store = writable<CupcakeStore>({
  toppings: [],
  tasty: true,
});

const cupcakeStore = _baseStore(
    store, 
    ({ subscribe, _syncToLocal }) => {

      /* cupcake things */

      _syncToLocal('myCupcake');

      return {
        subscribe,
      };
});

export default cupcakeStore;

You might have noticed I also added a _set function - this is just a more concise way of using the regular update method, so instead of writing:

update((store) => ({
    ...store,
    foo: bar    
}))

we can just write:

_set({ foo: bar })

It might not seem like a huge improvement, but it's quite useful when you want to update values multiple times in a function, for example.

Instances

So far pizzaStore has been a standalone store, assuming our app will only have one pizza. However it can be useful to have instantiable stores - imagine your app allows the user to create as many pizzas as they want. There are a couple of things needed:

Allow the creation of new pizzas through pizzaStore
Store all the pizzas somewhere, so we can access them elsewhere in our app.

The first issue is pretty simple, we simply export the function. For the second issue, one approach is to use a store of stores. It's a custom store that holds an object of string: PizzaStore:

// allPizzasStore.ts
type AllPizzasStore = Record<string, PizzaStore>;

function createAllPizzasStore() {
  const { subscribe, update } = writable<AllPizzasStore>({});

  function addPizza(name: string, pizza: PizzaStore) {
    update((st) => ({
      ...st,
      [name]: pizza,
    }));
  }

  return {
    subscribe,
    addPizza,
  };
}

const allPizzasStore = createAllPizzasStore();

export default allPizzasStore;

We then add our pizzas to this underlying store whenever they are created. In order to later get the store we need, we also pass in a name key to store it with:

//pizzaStore.ts
type PizzaWritableStore = {
  ingredients: string[];
  ovenTime: number;
  status: 'idle' | 'preparing' | 'baking' | 'ready';
};

export interface PizzaStore extends Writable<PizzaWritableStore> {
  addIngredient: (ingredient: string) => void;
};

export default function pizzaStore(name: string): PizzaStore {
  const store = writable<PizzaWritableStore>({
    // writable store data
  });

  const customStore = _baseStore(
    store,
    ({ _set, _syncToLocal, update }) => {

      // .. other functions like addIngredients, etc

      return {
        ...store,
        addIngredient,
      };
    }
  );

  // add it to the store of stores
  allPizzasStore.addPizza(name, customStore);

  return customStore;
}

And then we use it:

// Allow user to create new pizzas
<button 
    type="button" 
    on:click={() => pizzaStore("pepperoni")}
>
    Add pepperoni
</button>
<button 
    type="button" 
    on:click={() => pizzaStore("tuna")}
>
    Add tuna
</button>

// And in some other component:
<script lang="ts">
    import allPizzasStore from "./allPizzasStore";

    let pep = $allPizzasStore.pepperoni
    let tuna = $allPizzasStore.tuna
</script>

<div>
    The pepperoni is {$pep.status}
    <button 
        type="button" 
        on:click={() => pep.bake()}
    >
        Bake it
    </button>
</div>

Any pizza we now create is accessible in the rest of the app via the allPizzaStore!

Wrapping up

Hopefully this demonstrates how flexible and useful stores can be in Svelte. Often, a simple store may be enough, but when necessary, we can easily adapt our stores to fit our needs.

Thanks for reading!

How to build a Concurrent & Resilient Service in Elixir

José Diogo Viana — Thu, 16 Feb 2023 17:05:42 +0000

During the last 12 months, more or less, I had to build 3 different services that were highly concurrent and required some degree of resilience and fault tolerance. These services had different requirements and did completely different things, but after the third one, I noticed that I'm pretty much using the same recipe for the structure. That's what I'll talk about in this blog post, and at the end, I expect to have given enough insight into this recipe, so that you can also follow it in your next project or even make it better (if you have suggestions feel free to contact me at one of my socials).

Resilience & Fault Tolerance

In this title, I mention Resilience and there's also Fault Tolerance in the first paragraph. After all, what are these concepts about? I learned about them, more in-depth, back when I was getting my Master's Degree in Distributed Systems, and I've had them at the back of my mind every time I need to build a system that requires high availability and correctness.

If you want to be Fault Tolerant, then as the name says, it needs to be able to deal with possible failures that may occur. This ranges from writing defensive code, and being prepared to handle these errors, to the ability to maintain functionality when portions of a system break. The Wikipedia definition has the following analogy:

An example in another field is a motor vehicle designed so it will continue to be drivable if one of the tires is punctured, or a structure that is able to retain its integrity in the presence of damage due to causes such as fatigue, corrosion, manufacturing flaws, or impact.

The example above can also enter the realm of Resilience, but I see it more as the ability to recover from a partial or total crash. If you have a running service that shuts down for some reason, you may not need to do anything special when restarting, but for some occasions, it can be necessary to recover the previous known state and resume from that. These systems are very common when the previous state matters.

I think this Post from Uwe Friedrichsen makes a good summary of what Fault Tolerance is, and it also makes a comparison between both concepts, as they have a lot in common.

Please don't get confused or think I'm talking about Byzantine Fault Tolerance, as that's a matter for actual Distributed Systems, and not for Concurrent ones. Concurrency is also not the same thing as Parallelism, so what is Concurrency after all?

Concurrency

In computing, concurrency refers to multiple things that overlap in time, so that one starts before the other finishes, and they "fight" each other over CPU time. Parallelism is multiple things happening at the same time. Multiple processes or threads on a single time-sliced CPU might exhibit concurrency but not parallelism. Parallelism is about multiple tasks that literally run at the same time on hardware with multiple computing resources like a multi-core CPU. As per Rob Pike:

Concurrency is about dealing with lots of things at once. Parallelism is about doing lots of things at once.

In this blog post by Manning Free Content Center there's a great explanation including two images that inspired me to do my own below. You should definitely read it if you're interested in the matter. (Or if you just want the 5 year old explanation check Joe Armstrong's Concurrent and Parallel Programming.)

As for Distributed systems, well, they're distributed... Who would have guessed? 😅 This means that a distributed system is one in which components are decoupled from one another. Distribution also introduces issues of consistency and separate failure domains (that's where those Byzantine Faults enter). In a non-distributed system, consistency is often handled by that single machine and everything lives or dies together. If the hardware fully fails, then the service is completely down.

So here is a small summary just to consolidate knowledge before jumping on to the next phase of this post.

Concurrency (A robot only Singing and Dancing at each point in time).

Parallelism (A band performer Singing and Dancing at the same time).

Distributed (A full band performing together).

The BEAM & Elixir

As almost everybody at this point knows, Elixir compiles into BEAM (the Erlang Virtual Machine) byte code (via Erlang Abstract Format). This means that Elixir code can be called from Erlang and vice versa, without the need to write any bindings.

For the ones that don't know Erlang or its background, it's a language developed back in the ‘80s by Ericsson for better development and stability of telephony applications, and that was made open source in 1998.

This way, Elixir has all of the benefits of the battle-proven system that is Erlang, and it is possible to use the existing Erlang libraries with no performance penalty. Another strong point for Elixir is concurrency (you probably see where I'm going with the line of thought), which, in most languages, is a bit of a pain. Not only is dangerous, but it's also hard to achieve synchronisation. In Elixir it is effortless to create a new process, and it performs very well. It is a core feature of the platform and does not require a separate library. Note that Erlang/Elixir processes are not native processes; they are much more lightweight and are scheduled by the BEAM.

So, introductions aside, from this point I'm going to assume that you have at least the basic knowledge about the Erlang and Elixir ecosystems.

If you're reading this, there's a chance that you already built some Phoenix Apps and know how amazing it is to build upon it while keeping the concepts above somewhat abstracted under it. However, that's not why I'm here today. Have you ever needed to build a resilient and concurrent system, just like Phoenix, for example, inside your usual Backend (if you're into Monoliths) or as a stand-alone app (if you're into Microservices)? In the case you already have, you may see a lot of similarities between them, and it's those that I want to talk about and include in my recipe.

Taking advantage of Supervisors

Erlang has the Let it crash philosophy, and that's where Resilience and Fault Tolerance enter the scene. Don't get confused, the idea behind is not to let it crash and burn, but to let it crash and recover. Instead of having uncontrolled failures, Erlang turns failures, exceptions and crashes into tools that we can use and control.

The language comes with a lot of different tools, and you should definitely learn more about them. Here, I'll mostly prioritise Supervisors, but in this Medium post you can find more info on Processes, Monitors, and others.

Even though Supervisors come from Erlang, I want to clarify that from now on I'm talking of Supervisors on Elixir. Looking at the definition in the Elixir docs:

A supervisor is a process which supervises other processes, which we refer to as child processes. Supervisors are used for building a hierarchical process structure called a supervision tree. Supervision trees provide fault tolerance and encapsulate how our applications start and shut down.

The Supervisor process has the job of looking at the health of its child processes and acting accordingly to the specified configuration and strategy. These can range from configuring:

:shutdown: defines how a child process should be terminated.
:restart: defines when a terminated child process should be restarted.
:strategy: how the supervisor should act when a process terminates.
:max_restarts: the maximum number of restarts allowed in a time frame.

There are several more, and each of these has various options, but I feel that with the ones above you can already grasp what we can achieve with Elixir to quickly develop a Fault Tolerant and Resilient service.

Dynamic and Partition Supervisors

DynamicSupervisor: As the name implies, it's a supervisor that starts children dynamically. The Supervisor module (in Elixir) was designed to handle mostly static children that are started in the given order when the supervisor starts. A DynamicSupervisor starts with no children. Instead, children are started on demand. When a dynamic supervisor terminates, all children are shut down at the same time, with no guarantee of order. We'll see them in my simple framework for these services.
PartitionSupervisor: A supervisor that starts multiple partitions of the same child. Certain processes may become bottlenecks in large systems. If those processes can have their state trivially partitioned, in a way there is no dependency between them, then they can use the PartitionSupervisor to create multiple isolated and independent partitions. It's also possible to combine a PartitionSupervisor with a DynamicSupervisor (check this amazing post by AppSignal for a concrete example).

A simple Framework

We have the concepts in place, and we also know about Erlang/Elixir tools to manage processes, so it's time to jump into some actual code.

The following image tries to represent the base structure I've been using for these last 3 services I had to build/incorporate into an Elixir app. It all starts by starting a DynamicSupervisor under the main App Supervisor (usually where the Application code is). Then this DynamicSupervisor starts and manages all worker processes that should do all the heavy and concurrent lifting. Note that as stated above, the DynamicSupervisor can be replaced or mixed in with a PartitionSupervisor if the use case requires (in my situation, I haven't had to use them yet, but they are next in line if there is any bottleneck in the process).

With this approach, you can also guarantee that an unhealthy Worker Process won't affect the others, as they are separate entities. If you want to ensure synchronization between them, then adding a new Manager process can be an easy solution (check my previous blog post on Simulations with Elixir for a quick example). For the hard way but more robust, you probably need to look into Consensus Algorithms, but that's more in the realm of Distributed and not Concurrent only systems.

You probably noticed a new term in the image, theETS or Erlang Term Storage. An ETS allows us to store any Elixir term in an in-memory table and use it as a cache mechanism (learn more about them). I usually use this table to keep track of Worker Processes' PIDs and other metadata about each one of them (depending on the use case and the amount of information that each Worker needs to know about itself, or others). Be careful not to use it as a cache prematurely! It's an in-memory store only! For persistent solutions, we'll see other possibilities, below.

Usually, I also add some client functions to the DynamicSupervisor module, in order for Worker Processes to be able to update info on the ETS or trigger the start/stop of other Workers if needed. These functions can be abstracted into a separate Module if this one starts to grow indefinitely, and you feel they should be separated.

defmodule MyApp.MyModule do
  require Logger

  use DynamicSupervisor

  def start_link(init_arg) do
    with {:ok, pid} <- DynamicSupervisor.start_link(__MODULE__, init_arg, name: __MODULE__) do
        # Extra logic here if needed
        {:ok, pid}
    else
      _error ->
        Logger.error("Error starting MyApp")
    end
  end

  @impl true
  def init(_init_arg) do
    DynamicSupervisor.init(strategy: :one_for_one)
  end

  def start_worker_instance(args) do
    spec = {MyApp.MyModule.Worker, args}

    with {:error, error} <- DynamicSupervisor.start_child(__MODULE__, spec) do
      Logger.error("Error starting new Worker: #{inspect(error)}")
    end
  end

  def stop_worker_instance(some_id) do
    with {:ok, pid} <- get_pid_worker_instance(some_id) do
      Worker.stop(pid)

      {:ok, :closed}
    end
  end

  def get_pid_worker_instance(some_id) do
    case :ets.lookup(:my_ets_table, some_id) do
      [] -> {:error, :not_found}
      [{_some_id, pid, _rest}] -> {:ok, pid}
    end
  end

  def delete_worker_from_ets(some_id) do
    :ets.delete(:my_ets_table, some_id)
  end

  def insert_worker_in_ets(some_id, pid, rest) do
    :ets.insert(:my_ets_table, {some_id, pid, rest})
  end

  # ...
end

With the piece of code above, we already have a very basic supervision tree that can help us achieve a lot of things without much hassle. If a Worker Process dies, it's restarted, and the same thing for the DynamicSupervisor itself. Don't forget to start this Supervisor under the Application, or another already there.

The code for the Worker usually looks something like this:

defmodule MyApp.MyModule.Explorer do
  require Logger
  use GenServer, restart: :transient

  def start_link(args) do
    {:ok, pid} = GenServer.start_link(__MODULE__, args)

    MyApp.MyModule.insert_worker_in_ets(args.some_id, pid)
    send(pid, :start)
    Logger.info("Starting #{args.some_id}.")

    {:ok, pid}
  end

  def stop(some_id) do
    GenServer.stop(some_id)
  end

  @impl true
  def init(init_arg) do
    # new_state = some kind of computation on init_arg

    {:ok, new_state}
  end

  @impl true
  def handle_info(:start, state) do
      # start work
  end

  # ...
end

Note the usage of use GenServer, restart: :transient. This piece of code tells the Supervisor to not restart this process if it stopped in a normal/controlled way. Only in a case of a crash, it will be automatically restarted. (You can then play around with other options.)

Going the Extra mile

I want to reiterate that the approach, or "recipe" as I called it previously, is as simple as you can get in order to start. There are obviously a lot of extra things you can add to make your system even more robust to faults. Here are some of them, that I've used in the past.

Using a `:dets` and other disk storage mechanisms

Imagine the situation that our service is indexing Events that happen in a separate place. If the service completely fails, when restarting we want to resume/continue indexing in the place where we left off before the downtime. This way we won't have any duplicated events, and won't miss others while down.

We can achieve this with the help of disk storage-based mechanisms, like using a :dets or the Database itself. A :dets is very similar to the :ets, but it's disk-based instead. It features a lot of helper functions that you can find in the docs, including ways to quickly turn it into an ETS, if we want.

IMPORTANT NOTICE: if you use services like Heroku or Fly.io, relying on this file approach may not work! Changes to the filesystem on one machine may not be propagated to others and are not persisted across deploys and/or restarts (depending on service specifics, they don't work the same way).

To avoid the problem above, you can rely on your dedicated Database or other services like Redis to keep track of it. The start_link code for the Supervisor would look like this now:

def start_link(init_arg) do
  with {:ok, pid} <- DynamicSupervisor.start_link(__MODULE__, init_arg, name: __MODULE__) do
    :dets.open_file(:my_dets, [])

    restore_previous_state()

    {:ok, pid}
  else
    _error ->
      Logger.error("Error starting Dora")
  end
end

defp restore_previous_state do
  if :ets.whereis(:some_pid) == :undefined do
    :ets.new(:my_ets, [:set, :public, :named_table])
  end

  case :dets.match_object(:workers, {:_, :_, :_}) do
    [] ->
      MyApp.Workers.list_workers()
      |> Enum.each(&start_worker_instance(&1.some_id, &1.extra_info))

    list when is_list(list) ->
      Enum.each(list, fn {some_id, extra_info} ->
        start_worker_instance(some_id, extra_info)
      end)

    _ -> Logger.error("Some unexpected error happened!")
  end
end

Note that it's also required to add/remove workers' entries from these tracking systems when a new Worker Process is started/stopped with success.

You can check our Dora, the TipsetExplorer repo with this approach, as one of the projects we submitted for the FVM Space Warp Hackathon.

Process Pool

If your use case requires a lot of Worker Processes, and you don't want to flood the BEAM with new processes, you should definitely go with a Process Pool solution. You can easily exhaust your system resources if you do not limit the maximum number of concurrent processes that your program can spawn. The solution is to use a set of workers (processes) to limit the number of connections instead of creating a process for every individual action. For this, I would suggest two different approaches:

Poolboy: widely used lightweight, generic pooling library for Erlang that addresses this issue. This Elixir School guide is very helpful to get started quickly.
Custom solution: if Poolboy doesn't fit well within your constraints, you can always build your custom solution. For a previous personal project, I took the opportunity to build my custom solution, not because I had special requirements, but because I wanted to try my own thing. You can find the code for this custom pool here. The file code I linked is a "special type" of Worker Process, under the DynamicSupervisor supervision tree, but only keeps track of the pool and other Process management-related things.

Different Handlers that can be plugged and played

Suppose we continue our Indexing example, highlighted in the persistent-store mechanisms chapter, and we have different ways to listen/query for those events being emitted. We could have for example a queue/socket for passive listening, but the other service could also feature an HTTP API that we could poll.

This way if we want an even more fault tolerant and resilient service, we can make our Worker Processes use different Handlers to interact with the external system. So, in case we start listening to events via a socket/queue and the external service has a failure on that section, we can switch to automatically polling it through the HTTP API. This approach requires some overhead to manage which handlers are being used, and when to switch between them.

Wrapping up

We've now reached the end of this post... It was probably a bit too technical, but I tried to explain all base concepts in a way that any developer, with different degrees of experience, could understand the problem and the proposed approaches. If you have any suggestions to add under the Extra mile section or even the base Framework, feel free to reach out!

As hinted in the previous sub-section, there are a lot more things to be explored and looked at according to your needs. I don't believe this is a hammer, and all my problems are nails, and you shouldn't believe it also! However, I think it should give a good place to start for personal/side projects.

Image from: https://raspstephan.github.io/blog/hammer-and-nail/#

And as always, thank you for taking the time to read this, and see you in the next one 👋

Genetic Algorithms to optimize an Asset Portfolio

José Diogo Viana — Fri, 23 Dec 2022 21:54:58 +0000

On the last weekend of October, we (finiam) participated in ETHLisbon, an Ethereum-related hackathon, and our submitted project, Matusalem, resembled a pension fund scheme but only for Crypto Assets - I won't dwell here into much details of the project, as we'll probably have a dedicated blog post explaining it.

So, why am I talking about an Ethereum hackathon and Genetic Algorithms? Well, by the title of this post, you can probably guess why. For the last few weeks I had been reading Genetic Algorithms in Elixir by Sean Moriarity, and I was hooked on giving them a try in a use case outside the book. Fast forward to ETHLisbon, and our idea for the project, we found a place where Genetic Algorithms could be a nice path to explore, trying to figure out from a set of Crypto Assets which ones should make it into our pension fund portfolio, and I became responsible for it.

The base idea

Forgetting the details of how the whole thing would work, in this blog post, we'll focus only on the process of calculating the percentages of the funds we should allocate to each asset from a given list, using Genetic Algorithms.

Starting from the simplest use case, imagine we have 4 Assets to choose from and invest the money. Let's call them assets A, B, C, and D. Supposing the table below represents their expected returns and associated risk (1-7).

	Returns (%)	Risk Level
A	4%	1
B	2%	3
C	8%	7
D	5%	4

If we wanted to calculate the percentages we should invest in each asset, just by looking at the expected returns, it's clear that we should put all eggs in the basket C. However, things are just not that simple regarding investments, as we can understand that different assets have different risk levels. Normally, the higher the expected return, the higher the risk, but that's not always the case. So, our final goal, for this hackathon, was to optimize for a portfolio allocation that would maximize returns while bearing minimal risk. How can we do that? (At least in a mathematical way).

Sharpe Ratio

Here comes in Sharpe Ratio. We can evaluate the investment performance based on the risk-free return. The risk-free return is the return of an investment with zero risks for the investor (for example US Government Bonds, etc.).

A Higher Sharpe metric is always better than a lower one because a higher ratio indicates that the portfolio is making a better investment decision.

Since Genetic Algorithms are an optimization problem, and I'll explain further down how they work, we can use Sharpe Ratio as the metric we want to optimize. It takes into account expected returns and also associated risks. But how does it work, then?

The image is from WallStreetMojo's explanation of the Sharpe Ratio, and I recommend reading it if you're curious about these subjects.

If you're not familiar with Sharpe Ratio or Statistics in general, you may be asking how does the formula above take into account the assets' risks? Well, that part is handled by the portfolio's standard deviation.

Portfolio Standard Deviation refers to the volatility of the portfolio, which is calculated based on three important factors that include the standard deviation of each of the assets present in the total Portfolio, the respective weight of that individual asset in the total portfolio, and the correlation between each pair of assets of the portfolio.

This way, if a portfolio has a high standard deviation it means that the risk is high, it has more volatility and therefore not so stable returns. A portfolio with a low standard deviation has less associated risk.

We will see below how we can calculate the standard deviation for a portfolio and use it inside our Genetic Algorithms' fitness function. But what is a fitness function?

Genetic Algorithms

As stated above, Genetic Algorithms are used to solve optimization problems. An optimization problem is a problem of finding the best solution from all feasible solutions. Usually, the solution to that problem is hard to obtain using conventional mathematics, requiring a lot of computation (brute force).

One example of an optimization problem is the Knapsack problem.

Genetic Algorithms help us overcome this brute force approach, by using certain heuristics that enable us to find a "good enough" solution, but not necessarily the best one.

Despite the cool name, how does a Genetic Algorithm actually work? It tries to mimic evolution the same way it happens in nature. Concepts like Natural Selection are known by almost everyone, and if you're reading this post, you're probably familiar with it. Individuals with certain variants of a trait tend to survive and reproduce more than individuals with other less successful variants. This gives that phenotype (with valuable traits) a reproductive advantage and may become more common in a population - survival of the fittest. A key part of this is also Sexual Selection, where this form of selection, together with Natural Selection, means that some individuals have greater reproductive success than others within a population, for example, because they are more attractive or prefer more attractive partners to produce offspring. A Genetic Algorithm takes these concepts and runs a sort of simulation for an evolution of a population.

In his book Genetic Algorithms in Elixir, Sean provides a simple scheme like the one below, that summarizes really well the different components that take part in these algorithms.

What does it mean, and how does it relate to Darwin's theories? Let's imagine we have a population for a species of Unicorns and follow the diagram above.

Initialize Population: In our case the starting population doesn't evolve out of a different species, it just gets "initialized randomly" (we'll see how we can represent a population later). That means that we start with a population of individuals we know are unicorns (four legs, a spiralling horn projecting from its forehead, etc), but their traits (genotypes) would be randomized across the entire population. Traits like height, horn size, muscle density, etc. A good starting population for these algorithms should be random enough (but still inside valid boundaries) so that we don't simulate evolution with a starting bias. The population size also needs to be a finite number, for example, 100 individuals.
Evaluate Population: In this phase, we give each individual in our population a rating, accordingly to a pre-defined fitness function. After the rating process, all individuals are sorted by the rating they got. The best one is checked against the termination criteria. What are the termination criteria? It's a function that verifies if we have reached an optimal solution (it compares with a reference value) or could check if the rating is getting better each generation, if we've reached a specific number of generations, etc. If this function determines to not stop the process, the algorithm advances to step 3, otherwise, it returns the best individual and its rating (fitness function) providing us with the solution for the problem.
Select Parents: If the simulation continues to this step, then we must select the parents for the next generation. Selecting parents is basically following the idea stated in the Sexual Selection theory, and we group the best individual with the 2nd best, the 3rd with 4th, etc. for all individuals in the population set. This way, we plan to create children out of the best chromosomes, with the hopes of obtaining a fitter individual out of the match.
Create Children: After grouping each individual it's time to generate their children. To create children out of each pair of parents, we'll use crossover. Crossover is analogous to reproduction. It’s a genetic operator that takes two or more parent chromosomes and produces two or more child chromosomes. So, supposing we have parents P1 and P2, we can combine them by splitting their chromosomes randomly, and obtaining four chromosome parts P1-1, P1-2, P2-1, P2-2. Children for these two individuals are the chromosomes P1-1|P2-2 and P1-2|P2-1. Repeating this crossover process for all parent pairs.
Mutate Children: Despite initializing the population to a seemingly random distribution, eventually the parents get too genetically similar to make any improvements during the crossover. In order to reach a proper solution, Genetic algorithms, also need to insert some entropy into the simulation, by provoking "mutations". Mutations to children's chromosomes should happen with a relatively low frequency, for example, a 5% probability of a mutation occurring. Mutations in Genetic Algorithms could range from shuffling individual genes', to swapping them entirely with a new value, etc. Too many mutations could add too much entropy to the algorithm, and it wouldn't get a proper result.
Start from step 2, and redo the process again, starting a new generation, until it reaches an optimal solution, triggered by the fitness function rating being high enough to reach the termination criteria.

I hope that this explanation has been enough to understand how a Genetic Algorithm works before we enter into the technical bits.

The Book's framework

When starting to read the book, after a more in-depth explanation of what I mentioned here (a lot of things in this post are from there, also), the author guides the reader through the development of a framework that we can use for almost any Genetic Algorithm problem with minimal adjustments to its base logic.

Recalling the simple flow specified above, we can see that some things can be abstracted through some code, by using a defined API for what chromosomes and genes should be. To fully grasp how we got to the next pieces of code, you'll need to read the book, but the most important parts can be summarized into the following:

The definition of the Chromosome type.
- By using :genes as an Enum we can take advantage of all the existing functions. We can use complex structs to represent a single gene, but the set of them follows the same API.
- Defining other helpful fields like fitness, size and age that will help in the evaluation of the population.

defmodule Types.Chromosome do
  @type t :: %__MODULE__{
          genes: Enum.t(),
          size: integer(),
          fitness: number(),
          age: integer()
        }

  @enforce_keys :genes
  defstruct [:genes, size: 0, fitness: 0, age: 0]
end

The evolve function. This represents the flow above, where we get a population (list of Chromosomes), and iterate recursively over selection -> crossover -> mutation -> restart until the termination criteria are reached. I'll skip that part here, but this and the other functions remain almost the same across problems, but the book also shows different approaches.

def evolve(population, problem, generation, opts \\ []) do
    population = evaluate(population, &problem.fitness_function/1)
    best = hd(population)
    best_fitness = best.fitness

    IO.write("\rCurrent Best: #{best_fitness}")

    if problem.terminate?(population, generation) do
      best
    else
      generation = generation + 1

      population
      |> select()
      |> crossover()
      |> mutation()
      |> evolve(problem, generation, opts)
    end
  end
end

This way, the few things we need to worry about are providing the framework with an initial population and defining the terminate?(population, generation) function, which evaluates the termination criteria.

Using expected returns as Fitness Function

Recalling the problem stated above, for this project we wanted to know which Assets would make it into the portfolio using Genetic Algorithms. A good first approach for this, and one that is very simple, is just optimizing for the expected returns. Why? We can avoid the complex math related to Sharpe Ratio, and just worry about a singular variable: the asset expected return. We had already selected the assets we were going to use. Some of the expected returns, at the time of writing, were:

ETH Staking	Uniswap	BTC	Maker	Filecoin	Aave
4%	-0.0054%	-0.0017%	-0.0038%	-0.011%	-0.0009%

Note that this return would be in ETH tokens, and not Euros or USD. Looking at the table, the crypto winter is clear, and the only positive return is ETH Staking (you can check more here on how it works). It's not necessary to be a great Math genius to understand that the best solution to optimize our risk-reward portfolio, a huge chunk (100% basically) of the allocation should be placed under ETH Staking, looking just at expected returns. It's the only positive one 😅. We can use this knowledge to scope out the values presented as results by our Genetic Algorithm. If the optimal solution, when the termination criteria are reached, has higher allocations on negative expected return Assets, then something is not right. Pretty easy, no?

Let's start by modelling our problem in a structure that corresponds to the framework's API.

Gene: is a value ranging from 0 to 100, and represents the percentage of portfolio allocation to an asset. Initialized randomly.
Chromossome: has a list of 10 Genes. One for each different asset we want to analyse. We made sure that the initialized sum of the 10 assets was under 100 (we can't allocate more than 100% of our portfolio).
Population: 100 different Chromossomes. After each evolution round, we classify each Chromossome with the Fitness Function.
Fitness Function: For this first approach, the fitness of a given Chromossome is just multiplying the algorithm-predicted allocation with the Expected Return for that respective asset, and summing everything at the end. That would give us the global expected return for our portfolio. Example:

@expected_returns [0.04, -0.000054, -0.000017, -0.000038, -0.00011]
genes = [80, 5, 2, 3, 10]

fitness = 
 expected_returns
 |> Enum.zip(genes)
 |> Enum.map(fn {p, g} -> p * (g / 100) end)
 |> Enum.sum()

Termination Criteria: When the population age reaches 10000. There were other possibilities for termination criteria, but you should definitely read the book to understand more about this.

Well, after running everything a couple of times the results were just random (yeah, I ran a couple of times just to make sure the results wouldn't be different). What was wrong with it, then?

The culprit was the mutation function. I didn't talk about that in the previous paragraphs because I was relying on the function from the framework to run our algorithm. Big mistake. That function simply shuffled Genes around inside the same Chromossome, meaning that every time a mutation happened to a Chromossome in the evolution process, values from the initialization would just be shuffled, and not adding any entropy to the values themselves. That's why we always saw random values allocated for each Asset, it was just the random initialization.
- The solution for this required updating the mutation function. Instead of shuffling around the genes, the new mutation function should increase or decrease the current value randomly, so that values change throughout evolution. The code below solved that issue.
```
new_genes = 
  Enum.map(chromosome.genes, fn gene -> 
    range = floor(gene * 0.5)

    updated_value = gene + Enum.random(-range..range)

    if updated_value <= 0, do: 0, else: updated_value
  end)
```
However, problems didn't stop here. Now, we were having the best solution presented with 0 at almost every gene, including the ETH Staking one, which is the go-to asset. This one was more difficult to find out the root cause because there were two different reasons.
- Penalizing too much the very good Chromossomes. What's this about? Since that in the new mutation function, we're increasing the value randomly, sometimes the sum of all genes would be higher than 100 (something that we only account for in the initialization). Our fitness function had a small clause that if the sum of all genes was higher than 100, it would return 0 for that Chromossome's fitness, because it would be an invalid individual. With this, we were penalizing too much the Chromossomes reaching close to 100% allocation, effectively killing all of them until only smaller values remain in the population. The solution was to remove that clause from the fitness function and normalise the values if the overall sum went over 100.
- Penalizing too much the very bad Chromossomes. Looking at the mutation function there is the following line if updated_value <= 0, do: 0, else: updated_value. The problem with this was that if a gene reached 0 or below (because of randomly decreasing the old value), the gene would never "recover" from that 0 value, effectively locking most of the assets' allocations to 0. Updating the line to something like: if updated_value <= 0, do: Enum.random(0..2), else: updated_value fixed this problem, because it gives a small chance for that gene to have a different allocation value, and not just 0.

After these changes, we're getting pretty acceptable results where the Genetic Algorithm's best solution, presents ETH Staking with >90% allocations, and the other assets take home the rest 10%.

Using Sharpe Ratio as the Fitness Function

I've already explained what Sharpe Ratio is and why it's the final approach for this problem. TLDR, it also takes into account the inherent variation of Assets (associated risk) and not just their expected returns.

Recalling the Sharpe Ratio function from one of the images above, we need to know the portfolio expected returns, risk-free rate and the portfolio's standard deviation. We already know how to calculate the first two, we just need to calculate the standard deviation.

According to WallStreetMojo, the following formula is how you do it.

Where:

wX: allocated "weight" to asset X.
R(kX, kY): the correlation between asset X and Y.
σX: Standard Deviation for asset X.

We pre-calculated all of the correlations between Assets, and the Standard Deviation for each one. Implemented the function to calculate the full portfolio's Standard Deviation, and replaced the fitness function above with the complete formula for Sharpe Ratio.

The results were just bad, and nonsense all over again... The problem this time was that the time until the end of the Hackathon was running out, and there were other things in the project that needed to be done, so after a couple of hours of debugging it, we decided to move over.

It's a bit of a bummer finishing things this way, but at this point, the issue was probably in the pre-calculated values and not in the Genetic Algorithm itself, as we just changed the fitness calculation. The main thing for this blog post is the algorithms, and not Hackathon rushed statistics...

Wrapping up

Looking back at how things went, I should have finished reading the book before the Hackathon 😅, to reduce some of the inherent stress that comes with participating in one, instead of going insane debugging why my population doesn't evolve properly.

Despite not reaching a perfect or even a "working" solution, I learned a lot about tweaking these algorithms (after all, evolution is not that simple). We also ended up winning a prize of 1500$ awarded by Protocol Labs for the third-best project of the hackathon using one of their products (Bacalhau - cod in Portuguese, which comes from the extended name of Computing over Data). We used Bacalhau to run our genetic algorithms simulation in a decentralized network of servers, that automatically stores its results in IPFS.

Have a look at the code written during the Hackathon in our Github repo (not just the Genetic Algorithms code), and feel free to reach out if you have any questions. A small disclaimer, remember that it is all Hackathon code, and it gets a bit messy...

Thanks for reading, and see you in the next one 👋

On demand ISR with Next.JS and Sanity

David Lange — Sat, 10 Sep 2022 16:07:36 +0000

Static sites are great, right? You get excellent performance and can make the most out of CDNs and caching. Frameworks like Next.JS have made it quite easy to create and deploy static content, and you can mix it up with SSR (server side rendering) when needed.

SSG (static site generation) has traditionally had a pretty big downside though - when you need to update something, you have to rebuild the whole thing all over again. With Next.JS though, we can mitigate that problem with ISR (Incremental Static Regenaration). Basically, ISR adds functionality to regenerate specific pages, and version 12.1 of Next brought us the highly requested on demand ISR.

This means a static Next site that fetches content from a CMS can now get updated on demand, (re)building only the parts that actually need to change.

So I decided to investigate how I could update a static site connected to Sanity CMS. For those of you who don't know, Sanity is an open-source CMS - it's flexible and pretty easy to get started with, and a great choice if you want to just host some content.

On demand revalidation

First, here's the example for ISR in the Next.JS docs:

// pages/api/revalidate.js

export default async function handler(req, res) {
  // Check for secret to confirm this is a valid request
  if (req.query.secret !== process.env.MY_SECRET_TOKEN) {
    return res.status(401).json({ message: 'Invalid token' })
  }

  try {
    // this should be the actual path not a rewritten path
    // e.g. for "/blog/[slug]" this should be "/blog/post-1"
    await res.revalidate('/path-to-revalidate')
    return res.json({ revalidated: true })
  } catch (err) {
    // If there was an error, Next.js will continue
    // to show the last successfully generated page
    return res.status(500).send('Error revalidating')
  }
}

The idea here is that you can send a request to a api/revalidate endpoint, and then call the revalidate method on the response object, passing in the path you want to update. Pretty simple right?

Sanity custom actions

So how do we call that revalidate endpoint from Sanity?

One option is using Sanity's webhooks, and there's already an article on that right here!

However, you might need a little more flexibility and control - for instance you might not want to trigger rebuilds on every single update. Or maybe you need to send requests to different frontends at different times (like when you have a staging site and a production site).

In these cases, you can use custom Sanity actions. Basically, these let you add buttons to the studio editor.

You can find a good starter on adding actions in Sanity's docs here, so I'll skip to the action itself:

export function MovieUpdateBlog(props) {
  const [status, setStatus] = useState("pending");

  const label = useMemo(() => {
    switch (status) {
      case "success":
        return "Updated";
      case "error":
        return "Something went wrong";
      default:
        return "Update blog";
    }
  }, [status]);

  if (props.type !== "movie") {
    return null;
  }

  return {
    label,
    onHandle: async () => {
      // this gets called when the button is clicked
      try {
        await revalidatePath(`/movie/${props.published.slug.current}`);

        setStatus("success");
      } catch (err) {
        setStatus("error");
      } finally {
        // Signal that the action is completed
        props.onComplete();
      }
    },
  };
}

First, we check what the document type is - in this case I only want this action to apply to "movie" documents. When the button is clicked, the onHandle function runs, and we pass in the document slug to a revalidatePath function:

async function revalidatePath(path: string) {
  try {
    const endpoint = new URL(FRONTEND_API_URL);

    endpoint.searchParams.append("path", path);
    endpoint.searchParams.append("secret", SECRET_TOKEN);

    await fetch(endpoint.href);
  } catch (err) {
    console.error(err);

    return null;
  }
}

This sends the request with the updated path to our frontend's revalidate endpoint. Success!

No, it's not so simple

In most cases one updated piece of content will require other changes elsewhere. Imagine a static site about movies - the homepage will show links to each movie's page, and on that page there's a list of the movie's cast, who also have their own page:

Fig.1: complex diagram clearly demonstrates how I could have been an architect

If we update Movie #1's name, we'll need to rebuild its page and the homepage, or else it would still show the old name. And if we update Person A's name, we need to also update the page of any movie that mentions them.

This will always depend on how your site is structured, but in this case we can deal with this logic within the action we just created:

await revalidatePaths([`/movie/${props.draft.slug.current}`, "/"]);

async function revalidatePaths(paths: string[]) {
  try {
    const endpoint = new URL(FRONTEND_API_URL);

    endpoint.searchParams.append("paths", paths.join(","));
    endpoint.searchParams.append("secret", SECRET_TOKEN);

    return fetch(endpoint.href);
  } catch (err) {
    console.error(err);

    return null;
  }
}

We're simply passing the movie's path and the / homepage path. We then have to deal with multiple paths on revalidation endpoint:

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  // Check for secret to confirm this is a valid request
  if (req.query.secret !== process.env.REVALIDATION_TOKEN) {
    return res.status(401).json({ message: "Invalid token" });
  }

  try {
    const paths = (req.query.paths as string).split(",");

    for(let i = 0; i < paths.length; i++) {
      await res.revalidate(paths[i]);
    }

    return res.json({ revalidated: true });
  } catch (err) {
    // If there was an error, Next.js will continue
    // to show the last successfully generated page
    return res.status(500).send("Error revalidating");
  }
}

So what about updates to Person pages? For that, we need a new action with a little more logic:

export function OnUpdatePerson(props) {
  const [status, setStatus] = useState("pending");

  const label = useMemo(() => {
    switch (status) {
      case "success":
        return "Updated";
      case "error":
        return "Something went wrong";
      default:
        return "Update blog";
    }
  }, [status]);

  if (props.type !== "person") {
    return null;
  }

  return {
    label,
    onHandle: async () => {
      try {
        const dependantMoviesReq = await fetch(`${SANITY_ENDPOINT}?query=*["${props.id}" in castMembers[].person._ref]`);

        const { result } = await dependantMoviesReq.json();

        const movieSlugs = result.map((movie) => `/movie/${movie.slug.current}`);         
        const personSlug = `/people/${props.published.slug.current}`;

        await revalidatePath([personSlug, ...movieSlugs]);

        setStatus("success");
      } catch (err) {
        setStatus("error");
      } finally {
        // Signal that the action is completed
        props.onComplete();
      }
    },
  };

To know what movies mention the updated person, we need to query Sanity. And if you're wondering what on earth this is:

`${SANITY_ENDPOINT}?query=*["${props.id}" in castMembers[].person._ref]`

it's GROQ, Sanity's custom query language. I would probably just use GraphQL, but I thought it would be interesting to try out GROQ for this demo.

Anyway, the query returns an array of movies that refer to the person. So we just get those slugs and add the current person slug, and pass it all to revalidatePath. And now we can update all the affected pages with just one click. This kind of logic will really be specific to each case - you should also change what paths you update based on the specifc data that was changed - an update to the person's bio might not require rebuilding pages that only contain their name.

Adding new data

Does this work when creating new pages? Yes, but there's an important detail you might miss. In the movie page scenario, I'm using getStaticPaths + getStaticProps:

export const getStaticPaths: GetStaticPaths = async () => {
  const { allMovie } = await getAllMovies();

  const paths = allMovie.map((item) => ({
    params: {
      slug: item.slug.current,
    },
  }));

  return {
    paths,
    fallback: true,
  };
};

If fallback is false, it won't work - the result of getStaticPaths would be the same as at build time, when the new content didn't exist. Setting it to true ensures it gets called again.

Some things to consider

With that, we have a much smarter way of updating static pages. Still, there are some things you should consider before you start ISRing everything:

Serverless function limitations. There is usually some execution timeout on these (you can check out Vercel's here), so there's a limit to how many paths you can revalidate.
Are you using a cached endpoint? For instance, Sanity provides a live endpoint and a cdn version. If you use the cdn endpoint in your build, it might not get fresh data when revalidating.
You will need to setup CORS for your endpoint to accept the requests once it's deployed. There's a guide on how to do that here.
Don't do this in the api/revalidate endpoint:

paths.map(async (path) => await res.revalidate(path));

await Promise.all(paths);

This was my first approach, and though it worked fine in local builds, when I deployed it to Vercel it would just fail. Basically you have to revalidate the paths sequentially, so Promise.all is not an option.

Wrapping up

On demand ISR is still a pretty recent feature, and I wouldn't be surprised if it changed somewhat in the near future. For now though, it looks like it could make static sites a lot easier to manage, and cut down on build times.

Spotlight effect with JS and CSS

David Lange — Fri, 29 Apr 2022 16:00:45 +0000

Spotlight effects can be a pretty creative way of revealing content on your website. And they're surprisingly easy to create with a little JS and CSS!

The basic idea

TLDR; Use CSS radial gradient backgrounds, and then use JS to track the mouse movement and move the center of the gradient.

There are other, more complex ways of doing this - you could use canvas or external libraries - but I find the simpler approach is a good starting point and should work well enough for most use cases.

Let's get started:

#spotlight {
  position: fixed;
  opacity: 1;
  width: 100vw;
  height: 100vh;
  top: 0;
  left: 0;
  pointer-events: none;
}

<div id="spotlight"></div>

const spotlightEl = document.querySelector("#spotlight");

function handleMouseMove(event) {
    const { clientX, clientY } = event;

    spotlightEl.style.background = `radial-gradient(circle at ${clientX}px ${clientY}px, #00000000 10px, #000000ee 350px)`;
}

document.addEventListener("mousemove", handleMouseMove)

And...that's pretty much it.

The #spotlight div is positioned so that it always covers the entire viewport. We then attach an event listener to listen to the mouse's movement, and set the X and Y directly in the radial-gradient CSS value.

The #00000000 10px, #000000ee 350px part basically means we want a gradient that is transparent at it's center, and at 350px from the center is a slightly transparent black. That's how the gradient center reveals what's beneath.

Making it configurable

So far we have a basic spotlight, so let's go ahead and improve it.

We can create a Spotlight class that allows you to easily set up new spotlights and pass in some options:

toggleEl is the id of the element you want to use as a trigger to toggle the spotlight.
innerRadius and outerRadius set the spotlight size.
outerColor sets the background color (try passing in red for a truly bleeding edge experience).

The class also provides methods for switching the spolight on and off.

Making the light feel more natural

To make it a look a little more natural, there are a couple of added touches.

First, there is a slight delay in the spotlight movement. This adds a sense of weight to the light, like we're dragging it around with our mouse.

handleMouseMove(event: MouseEvent) {
    setTimeout(() => {
      this.updateEl(event.clientX, event.clientY);
    }, 50);
}

Also, you'll notice the light "pulses" slightly - continually increasing and decreasing.

But how do we get it to work?

My first instinct was to animate the gradient size, but that's not actually possible in CSS! Next I thought of somehow changing the outerRadius value in steps inside a function called in a setInterval, but that turned out to be a really stupid idea. Finally, I came up with the simple solution - animate the scale of the entire spotlight div with a CSS animation.

@keyframes pulse {
  0% {
    transform: scale(1);
  }
  100% {
    transform: scale(1.1);
  }
}

this.el.style.animation =
        "pulse 3s ease-in-out infinite alternate forwards";

This way, we don't need to touch the actual gradient, we just animate the entire "canvas", stretching the gradient in the process. I also used CSS animations for the "switching on/off" part.

Wrapping up

Obviously, you shouldn't be using spotlights in every single page (please don't!). But when used right, they're a nice way for users to feel like they're discovering some cool concealed content for the first time.

Thanks for reading!

Developer Productivity

Davide Silva — Wed, 13 Apr 2022 15:42:01 +0000

I like to think that I am a productivity oriented person. If there is a faster or more efficient way of doing a task, I tend to incorporate that as part of my workflow. This can be small things like being able to jump directly to a certain application without having to do the CMD+Tab dance or being able to run a test without having to leave my editor.

As a long time Vim user, I also like to avoid using my mouse as much as possible.
Therefore, my current development setup is optimized so that I can perform most of my tasks using only the keyboard. I'll try to be as much editor agnostic as I can because it is not so much about a specific editor configuration, as it is more about the general workflow, but the examples I'll give will be in Neovim as that is my preferred text editor.

What to improve

There are a few things that I consider any developer should focus in order to improve their development workflow:

Basic file navigation
Jump to definitions or references
Run a test

Basic file navigation

Being able to quickly navigate a codebase, of any size, is a crucial skill for any developer. There are basically two phases of understanding a new codebase. There's the "everything is new and I don't know where anything is" phase and the "I know this codebase like the palm of my hand" phase. The method for navigating a codebase should also change with your increased understanding.

Tree based navigation

When we first start navigating a new codebase, we can make some educated guesses of where some of the logic is but we don't yet know exactly how the files are called or how the project is structured. If we are using a framework, we can expect some structure already, but this is not true for all the cases.
Surprisingly enough, the best way to become more familiar with the codebase is by looking through it. Seeing where things are in relation to another, how is the folder structure organized, any filename that might be worth having a deep dive, etc.

This is nothing new as file tree explorers have been a crucial part of most text editors, either natively or via a plugin, for a while. However, one feature that people seem to not know exists, is being able to show the location of a file in the tree.
Let's say that we need to make a change but we don't know exactly where the actual logic is. We know of a place to start but, as we are still very new to the codebase, we need to start jumping to function definitions or opening other files until we finally arrive at our destination. We finally found the file at the end of the chain but now we realize that our file explorer is closed and we have no idea of where we are. Most file tree explorers have a functionality that allows us to open our tree but instead of showing the root folder, opening the tree showing the current path until our current folder.
Voilà! We now know exactly where we are and the path we took.

Disclaimer: some editors may do this automatically or just require a keyboard combination or running a command. In others, you might have to configure something first.

Fuzzy finders

Tree based navigation is very useful and in some situations even superior to fuzzy finding. When we are looking through a folder to find a file but we don't really know its name or just when we want to have a sense of the size of the project. However, when we already know exactly where we want to go, having to navigate through the file tree is slower than just opening a prompt and typing the name of the file we want to open.
Fuzzy finding is just better because we don't need to fully type the name of the file we want. We can just start typing some of the characters that are part of the name of the file and we can usually get to the file very quickly.

Jump to definitions or references

Now that we know how to navigate our project quickly, we find a function that we don't know what it does or where it is defined. Having to stop our mental process to go find the file and scroll through to the function declaration is wasted time. Even more so if the function calls another function that calls another function.
What we really want is to put our cursor on top of the function call and, with a shortcut, jump to the function declaration. That way, we don't have to do the cognitive shift from reading code, to finding code, and back.
Another useful shortcut is showing all the places where a function is referenced. Let's say that we need to add a new parameter to a function. We will need to find all the places where the function is called to add the new parameter. Doing that so manually, is considerably slower than having our editor just compile a list that we can scroll through.

Thanks to the Language Server Protocol, we can now do just that. The installation varies from editor to editor, so I won't give any instructions here on how to install or configure it, but finding the appropriate installation instructions for your editor is just a matter of asking Google. The LSP also provides other functionalities that you should probably look into, like showing the documentation for a function or code formatting, and many more.

Run a test without leaving the editor

Here, at finiam, we follow a TDD approach when developing. Having to switch from the editor to the terminal to run the tests is wasted time, even if the terminal is incorporated in our editor.
Let's say we are writing a new test file. Having a shortcut to automatically start and run the test for us is a major time saver. We can even have different shortcuts to either run that specific test or run the entire test file or even the test suite.

There's another implicit advantage of being able to run the tests from our editor. If the barrier for running a test is just a simple keystroke combination away, we are more likely to run the tests when writing our code. If, for instance, running the tests is a process that forces us to switch from our editor to a terminal and typing the command to run the tests, we are more prone to skip them because we don't want to lose our focus. I am also guilty of this. I was once in a project that had a test configuration that forced us to start a Docker container to run the tests. I wasn't able to do my shortcuts to run the tests and, more times than I'd like to admit, I'd skip running the tests locally. The CI ends up having to do extra work when we open a PR for something that breaks the tests that we could have found locally, had we just run the tests. Eventually, we refactored that project enough that we didn't have to use Docker locally and I was back to using my shortcuts.

Conclusion

These small time save improvements end up compounding without you even noticing. The motivation is not to just be quicker at doing things but to shift the mental load of doing some repetitive tasks to muscle memory.
If every time we want to run a test or open a new file, we need to stop what we are doing and focus on doing that task instead, we lose track of that mental model that we spent so long building and we need to start again.

Take those boring tasks and make them so easy and so repeatable to do that eventually they become muscle memory and we don't even need to actively think about what we are doing.

DEV Community: Finiam

Building dapps with Starknet React

The good news

Getting started

Account

Provider

Contracts

Units

Starknet React

Connection and account

Network

Interacting with contracts

Read functions

Write functions

Single instance of a contract

Transactions

Wrapping up

Storing custom structs in Cairo 1

Custom Struct

Implementation

Write

Read

Wrapping up

Sign in with Metamask using LiveView

Why is it relevant?

How does it work?

Why Elixir and Liveview?

A look into the desired Flow

Getting our hands dirty

How to interact with Metamask from Javascript?

Wrapping up

A look into Cairo 1.0 traits

Cairo likes Rust

Traits

Using Generics

Final thoughts

Creating animated chickens with zdog

What it is not

What it is

Ease of use

The chicken

Animating things

Easing

Wrapping up

Exponential Backoff with Elixir

Getting started

GenServer for the stores

GenServer for the allowlist

GenServer for polling a store

Conclusion

Using custom Svelte stores

Custom stores

Wrapping the wrappers

Instances

Wrapping up

How to build a Concurrent & Resilient Service in Elixir

Resilience & Fault Tolerance

Concurrency

The BEAM & Elixir

Taking advantage of Supervisors

Dynamic and Partition Supervisors

A simple Framework

Going the Extra mile

Using a :dets and other disk storage mechanisms

Process Pool

Different Handlers that can be plugged and played

Further Reading

Wrapping up

Genetic Algorithms to optimize an Asset Portfolio

The base idea

Sharpe Ratio

Genetic Algorithms

The Book's framework

Using expected returns as Fitness Function

Using Sharpe Ratio as the Fitness Function

Wrapping up

On demand ISR with Next.JS and Sanity

On demand revalidation

Sanity custom actions

No, it's not so simple

Using a `:dets` and other disk storage mechanisms