DEV Community: The Agile Monkeys

Navigating the Double-Edged Sword of AI as a Software Developer

Abraham Romero — Sun, 13 Oct 2024 22:44:23 +0000

Introduction

Years ago, saving time in software development meant creating macros to avoid code repetition.. We needed to read books to understand how to program in a given language or for a specific machine.

We then turned to the internet for information, with YouTube videos, Udemy courses, and StackOverflow replacing traditional books.

Over time, we began using better IDEs like IntelliJ, which offers autocompletion and refactoring features that save us time.

Now, we have made a leap that is multiple magnitudes larger than ever before in terms of increasing development velocity. With the rise of LLMs, we now have a wide variety of tools that we, as developers, can use in our daily jobs. But, as Uncle Ben said, with great power comes great responsibility. Let's explore these tools together, let me tell you what I do, and what I think is coming.

Great Power...

You should already know all these names: ChatGPT, Gemini, Claude, GitHub Copilot, etc. You can find LLMs specialized for doing nearly everything you want to do nowadays.

The clearest advantage of these tools is increased velocity, such as rapid prototyping (AI can automate repetitive tasks, allowing developers to focus on higher-level design and problem-solving), faster debugging (AI tools can quickly identify bugs and suggest fixes, significantly reducing time spent in the debugging phase), or unit test development will be immediately noticeable.

But that's not all; you'll have access to a broader range of technologies. Using AI will reduce the learning curve by providing instant insights and resources on unfamiliar technologies. This allows you to quickly tackle projects outside your expertise with AI support.

LLMs like GPT-4 specialize in text generation, allowing you to write better messages, improve your emails, or generate documentation for your projects.

Great Responsibilities

As we just saw, using AI can help you do your job better. But it doesn't come for free; there are some risks.

You can't blindly trust AI. Over-reliance on AI can lead to complacency and reduced critical thinking. Additionally, AI-generated code may not always adhere to best practices or organizational standards, requiring vigilant oversight.

Relying on AI can stunt learning and hinder problem-solving skills, making it harder for developers to tackle challenges independently. Furthermore, if developers lean on AI tools instead of mastering foundational concepts, they may miss out on essential skills.

Security risks are another significant concern. AI-generated code may inadvertently introduce vulnerabilities, especially if not properly reviewed or tested. Cybercriminals can exploit these tools to automate attacks or create sophisticated phishing schemes.

How I Use AI in My Daily Job

In my daily workflow, I utilize several AI tools like ChatGPT, GitHub Copilot, or Perplexity AI. However, I approach their outputs with healthy skepticism. While AI can provide valuable suggestions and speed up certain processes, I always verify the results to ensure they align with best practices and meet my project’s specific requirements.

For example, I never copy and paste code from ChatGPT; I always try to understand the response and apply it to my code, test it, and check if it truly works. On the other hand, with GitHub Copilot, I accept many code suggestions, and that code is inserted directly into my code, but I always review it and modify it if needed.

This critical mindset allows me to leverage AI effectively without compromising the quality of my work. I also make it a point to stay engaged with the underlying concepts, ensuring that I continue to enhance my skills and understanding of the technologies I’m using.

What is Coming and How to Maximize the Advantages

Use AI tools, learn how to use them, keep learning, and focus on the personal aspects of your job, such as capturing requirements and team collaboration.

The future of AI in software development is bright, with advancements expected to make these tools even more powerful and intuitive. To maximize the advantages of AI, you should actively use these tools while also dedicating time to learn how to integrate them into your workflows effectively. Continuous learning is key; you should stay informed about the latest AI developments and explore new tools that can aid your tasks.

Importantly, it’s crucial to focus on the human aspects of software development that AI cannot replace. This includes capturing requirements accurately, engaging in effective team collaboration, and fostering open communication among team members. By maintaining a balance between leveraging AI and enhancing interpersonal skills, you can ensure you are not only effective in your technical role but also a valuable contributor to your team.

Conclusion

The integration of AI into software development presents a compelling mix of opportunities and challenges. While it can enhance productivity and expand skill sets, you must remain vigilant and engaged in your learning. Finding the right balance between using AI and mastering core software development practices is essential for long-term success.

Baby's First Contribution - A GitHub Flow for non-devs

Nick Tchayka — Tue, 25 Jul 2023 16:37:27 +0000

I'm sure that there was at least a single time where you wanted to change a word, a color, a font, a link, or something in a repo.

I'm sure that you are not afraid of doing the change, but of the process that it involves.

Using a text editor
changing code *gasp*
and… *dramatic orchestra sound* creating a pull request.

I know, this sounds scary af, but stay with me, I've streamlined the process for you so you can be doing the changes super easily.

Get to meet Git - Time Machine for code

If you use MacOS you probably know the Time Machine feature, which allows you to go back in time to retrieve a previous version of a file from the past. If you don't, well, apparently computers allow you to travel back in time 🤷 (at least digitally)

Git was one of the first software to popularize this concept, allowing to retrieve previous versions of any text file on a computer, saving all the changes that have been done to that file in what's called a log (like the log of an accountant, or the diary of someone).

The cool part of this, is that you can share this log with other people, so they can check the previous versions that you have done to your files. You can literally email them this log. (Remember all of this was in the early internet era).

What happens if two persons want to do changes to the same files? Git allows you to work in parallel, each timeline of changes is called a branch.

💡 Branches get out of the log, get it? Apparently programmers like bad jokes, but we keep denying it

They usually look like this:

Each circle is a change that someone has done, and the different colors represent the branches.

What if now we want to merge the changes from one branch into the main one? Let's keep reading.

GitHub - A Google Drive for Git logs

You could've been thinking already that emailing the Git log to everyone is a nightmare. Indeed it is, and teams have been doing so in the past.

Thankfully, nowadays we have GitHub, which makes everything super easy.

💡 Hub = place to share stuff, therefore GitHub = Place to share Git things.

GitHub is a platform that allows you to store the Git log of a project in a centralized place, and then it gives you tools so the team can discuss whether a change should be integrated into the main project or not (this is called a Pull Request).

If you don't have one, you should begin by creating an account in GitHub, and sharing the name with one of the team members so they can add you to the company team.

✏️ Note: In GitHub, open source projects are completely public, so if the project you want to contribute to is open source, you can skip the wait of getting added to the team.

GitHub is a very big tool that can have it's own article for an explanation. So we will talk only about making the change that you want.

Making the change

Let's suppose that you want to change a typo on the Docusaurus website of a project.

Open your web browser, and get into the GitHub repository that you want to make the change to.

Once you're here, press the . key in your keyboard. You should go from here:

To here

This new view is called Visual Studio Code. It is a text editor that devs use to code (and edit documentation).

Creating a branch

In order to be able to make a change properly, you need to create a branch.

First, think of a name for the branch (usually it is something like GITHUB-ISSUE-NUMBER/task-title for example 999/create-something-cool).

Now click on the branches button, which is the one saying main on the bottom left of Visual Studio Code:

Click on Create New Branch.

Type the branch name that you thought of, and press ENTER (if you minimize the window the text box will probably disappear and you will have to open this menu again)

It could be possible that it will ask you something about commits and stuff now, just click on Switch To Branch

Doing the actual changes

On the left side bar, you will see lots of folders and files. Locate the one that you want to change.

For our example, where we want to change something from Docusaurus, we will look into the website folder, which is where the documentation related files are usually located.

Let's say we saw a typo in the Getting Started guide and we want to fix it. We would have to navigate to website/docs/getting_started.md and open the file by clicking on it:

I'm going to change the first sentence of the section name Step 0.

💡 Usually technical documentation follows a format called Markdown. This is a way of formatting text in a super nerdy way, but it is a standard that most web tools understand. For example, when you format messages in Slack, you're using (something that is very close to) Markdown. Here you have a cheatsheet.

Once you save the file using Cmd + S. If you did it well, you should have a line marker where you did the changes:

Now to ensure that the changes are uploaded to GitHub, you have to commit them.

💡 Commiting means signing your changes with your name and a description. It is a way of marking who did what in a project.

On the top-left of Visual Studio Code, click on the thing that looks the USB logo, this is called the Source Control View:

Now, describe briefly the changes that you've made in the text box that says message. And click Commit & Push:

If the button went dark, and the message box got cleared, this means that the changes you made were uploaded to GitHub.

Now the last step!

Creating a Pull Request

If you put the mouse near where you see the text Source Control, some icons will appear. Click on the second one, the one that looks like a diagram with a plus sign. This will begin the Pull Request creation process.

This new panel should appear. Don't get into the details, you just have to do 3 things here:

Ensure that your branch is correctly selected.
Ensure that the into branch is main.
Improve the title, and add some lengthy description.

If everything is correct, click on Create.

Something like this will appear, click on Copy Link to copy the link to this Pull Request.

Now, just share the link with the team in a passive-agressive message, and let them review and merge the things that you did. Be proud of yourself!

Congratulations, you're officially a nerd programmer now!

Railway Event Processor: Safe integrations in Event Sourcing

Yeray Cabello — Mon, 15 May 2023 08:27:44 +0000

An integral default approach for the design and implementation of third party integrations in event-sourced systems.

⚠️The abstractions proposed in this paper will be soon implemented in booster, the algorithms proposed are proven correct in this repository. A solution to a real world problem with these abstractions will be soon shared.

Railway Event Processor

When in doubt, treat your system as distributed

The need to integrate with third-party services is prevalent in the software industry. Many cross-cutting problems present massive and unique challenges that require specialized focus to solve; some of them would be:

Making sure that transactional emails are sent to consumers without being filtered as spam requires such a degree of commitment that there are companies that do exclusively that.
Receiving payments online, interacting with the myriad of platforms available, and with the banks reaches a point of complexity that many payment providers become banks themselves to make the service more reliable.
Keeping track of interactions with customers has enough nuance to create an entire industry of Customer Relationship Management.
Logistics and chain of supply.
Providing customer support.
Authentication and authorization.

The issue with this is that, no matter how simple you think your application is, having any of those increasingly necessary integrations, effectively turn any system into a distributed one, and in the words of Leslie Lamport:

A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.

Any integration with these systems, services in your own company, or actually anything that happens outside the memory assigned on your program, like a database, presents four major challenges:

Orchestration.
Fault tolerance.
Concurrency handling.
Communication.

⚠️ This research looks to address Fault Tolerance and Concurrency Handling, but since they can’t be addressed without taking into account the other two, orchestration will be handled using the Process Manager Pattern, with some lessons learned from the Saga Pattern, and Communication will be solved via event streams and read models. It might catch the attention of the reader that, given the spotlight that message streaming platforms have in enterprise environments, to the point that seminal books as enterprise integration patterns deal almost exclusively with messaging, messaging is not used. As a matter of fact, the read models that will be used for scheduling can, and often should, be stored via message queues, but that extra step will be omitted for simplicity.

The language described in this paper has the aspiration to be a framework used as a bridge between engineering and business teams, so the degree of complexity of any given integration is visible and understandable by everybody involved.

First, the problem will be laid out in terms of the nature of integrations and why they are a frequent source of errors and technical challenges. Then the core concepts will be introduced, the Task, the Compromise, and the Transition.

All of these will be put together with an example in the Workflow definition.

Finally, a design will be provided for the abstraction of every kind of Task, with a formal proof of correctness for each. Together with the solutions, there will be proof of the unsolvable nature of the opaque integrations.

The solution design is built around the assumption of an event-sourced, or at least event-driven system, and it should be possible to apply these principles to such systems without any modification. A non event-oriented system can implement this pattern with relative ease as long as it follows the Command Query Responsibility Segregation principle. Beyond that, major concessions might be needed, forfeiting the correctness provided out of the box by this framework.

The end goal is to provide a safe structure to connect every step of these processes, so they can be built and maintained as a deterministic state machine.

Modelling systems as a workflow of tasks

Before delving into the concept of a task as the key component of our solution, it is essential to provide a brief introduction to the concept of a workflow as a means of orchestration. Details regarding this concept and the communication between tasks will be discussed later. At this point, it suffices to understand that any system orchestration situation can be modeled as a combination of tasks that run sequentially.

For instance, consider a series of tasks, labeled A to D, where task B is an action to be performed with information obtained from the result of A. Task A has the responsibility to start B. Then, when B completes, it will decide whether to continue with C, because customer support has to make a review, or D, which is a straightforward automatic process. Finally, either of them will emit an event with the outcome of the workflow.

The aforementioned configuration can be represented using a diagram as follows:

This example serves merely as an illustration of a possible task orchestration scenario. The crucial idea to grasp at this stage is that tasks will run in sequence, and that they are unlikely to show unexpected behavior, forming execution graphs of arbitrary complexity. These execution graphs provide a mental map of the system's orchestration, enabling efficient and adaptable process modeling.

More importantly, there is no hidden complexity in these diagrams, every possible outcome is represented in the graph.

The Task

An abstraction over a single step in a workflow, it is modeled after a state of a state machine, and all its possible results, including those of failure, will point out to a different task.

To-Do Item

They require one external action to happen: a user interaction, receiving a message from a queue, a webhook from a payment provider, or other similar interactions. Once that is received, a decision will be made to move to the next step.

State

A step in the workflow that will perform an action synchronously and, using the result of such action, will decide on the next step.

⚠️ Notice that neither of them are concerned with fault, the goal of the abstraction is exclusively to make a decision.

The transition

One key element of the Railway Event Processor is that there is no concept of a failed task. Even for technical faults, the inability to attempt the task has to be translated into a business outcome.

That is, a payment might be successful or fail because of a lack of funds, this means that the payment itself can be rejected, but the task is not concerned whether the result was desirable or not, but about that, there was one of two possible results, and that each of them will lead to a different task in the flow.

In this example, a successful payment will lead to the fulfillment of the order, and a failed one will lead to a new request for payment to the user, possibly with a retry counter, which will eventually lead to the order being canceled.

⚠️ A task that “fails” for business reasons it’s not considered a failure in this framework, as business failures (such as a payment rejected or not having permissions to perform an action) are abstracted as just one of several possible outcomes.

e.g. Performing a fraud check can determine whether the user is trustworthy or not. Both cases are just possible outcomes of the check: A result of potential fraud is not a failure but a successful verification that returned useful information.

The transition is represented as one of the following options:

A domain event that ends the workflow, this would be the outcome of the workflow.
The State that follows the decision.
The To Do Item that follows the decision.

⚠️ The domain event that ends the workflow can be used to chain parts of a more complex workflow.

Technical categories of tasks

External action required

The Task is waiting for the third party to continue the execution.

e.g. A webhook from a payment provider or a response sent by a human.

Idempotency enabled

The Task is executed against a service that provides a mechanism to uniquely identify an action. Said mechanism allows us to try to perform the action knowing that it will only impact the system once even if we have attempted it on the past or there is a duplicated process attempting the same action. This removes concerns for concurrency and fault tolerance, as it is possible to just attempt the action until we get an acceptance or rejection result and succeed in directing the workflow toward the desired next Task.

e.g. Many payment providers accept an Idempotency Key in their requests, so if we attempt a refund, and we generate a unique key per refund, we know we will never refund more than once.

Simply add the Idempotency Key to the request, any parallel executions, or retries due to failure to register the outcome will be ignored by the third party.

Optimistic lock enabled

This is aTask with the following characteristics:

Either the API of the third party offers:
- A way to attach a unique identifier to the action we want to perform.
- A query mechanism that allows filtering via the aforementioned unique identifier.
Or:
- The nature of the task makes it so that the action can be identified deterministically without an additional identifier.
- And the API offers a query mechanism that allows querying for that.
A multistep process that can be canceled at any time before it is completed.

This way, we can, before we even start processing it, detect whether there is another process trying to perform the same action or whether there was a previous attempt that faulted. This allows us to attempt the action and verify between steps again for faulty concurrent access. Finally, it is possible to run a cleanup after one process has been successful. That is, we virtually allow multiple processes to race concurrently, allowing only the first of them to write the result and canceling the others.

Pessimistic lock enabled

This is a Task with the following characteristics:

Either the API offers:
- A way to attach a unique identifier to the action we want to perform.
- A query mechanism that allows filtering via the aforementioned unique identifier.
Or:
- The nature of the task makes it so that the action can be identified deterministically without an additional identifier.
- And the API offers a query mechanism that allows to query for that.

Without a multistep process that can be canceled at any time, we must rely on a central place to lock any other processes out of attempting the operation.

That is, we won’t allow a second attempt to run if there’s an ongoing process.

Black boxed

This Task provides no ability to reliably determine if the operation is being performed by another process, or whether we are attempting it again from a previous failure.

For these processes, the beginning of an attempt is stored, so is the result, any attempt that starts, will check if there was a previous beginning recorded without a matching result, if that’s the case, a possible failure is detected, as it is impossible to know if the action was performed and its outcome.

This case would require compromising in case of failure. A decision needs to be made if it’s detected that there was a previous attempt that did not finish:

Have a human review it.
Ignore it and move on.
Retry it up to a certain amount of times.

Common language definitions

All these definitions, for the sake of simplicity, will be condensed from the perspective of the common language into these three terms, so when discussing the design of workflows that integrate third parties, they will get referred to in the following three categories.

To Do

This is the External Action Required type. An external service needs to reach to continue the workflow.

Trivial

Services that have an explicit idempotency mechanism, rare, but always preferable.

Integrating with this category poses no challenge both in terms of correctness and performance.

We can count on any action performed towards this service as that they will eventually complete.

Deterministic

This includes optimistic and pessimistic lock enabled third parties.

Integrating with this category poses a challenge in terms of correctness. The simplest approach is to consider all of them pessimistic and switch to the optimistic ones if there's a performance problem.

We can count on any action performed towards this service as that they will eventually complete.

Opaque

Black boxed.

These present an unsolvable challenge in terms of correctness, as we can neither guarantee a single execution nor failure (in the case of having a record of the attempt starting but not of the attempt outcome).

A business decision needs to be made in terms of what to do when failure is suspected.

Business concerns of a Black Boxed Task

This includes all the factors we need to take into account when making compromises over black-boxed tasks. This part of the framework is not formal, it aims to provide a checklist of factors to consider when making a decision, the decision itself is formal, as it will have an abstraction on top of it.

Visibility

How likely is the non-execution or failure of the task to be perceived.

Self-evident. By its own nature, the failure will be noticed. e.g. If a refund is never sent, the user will eventually complain.
Traceable. It’s possible to add reliable tracking mechanisms to follow up on the action being performed. e.g. The deduction of the loyalty points from a user that purchased a product using them fails. We would need to set an alert for it and have someone verify what happened.
Invisible. We can’t guarantee to know the outcome of a task. e.g. We cannot determine whether an email has been opened or not. Mechanisms like tracking pixels might be blocked by the email service, so it’s possible that an email that was read shows as unread.

Necessity

How critical is the step to the workflow

Must happen. The workflow can’t continue without this step. e.g. The payment must be confirmed before we start handling an order.
Should happen. The workflow can continue, but there’s a consequence if it fails. e.g. Bank account number validations are known to put back customers when they make a typo, so it’s desirable to allow them to continue and then reach out to them about a failed payment.
Nice to have. It’s not needed, but it is presumed that it adds some value to the product. e.g. Sending a satisfaction survey.

Concern

Who gets directly hurt if the task fails:

Own. e.g. There’s a failure when registering an order as sent, and it gets sent twice.
Third party. e.g. The delivery data for an order to a logistics company gets corrupted, and the provider needs to reach out to fix the issue.
Customer. e.g. A refund is not sent when it’s due.

Risk

The kind of consequences suffered:

Reputation. e.g. A customer gets charged twice for a product. Event after returning the amount, the customer might not come back.
Legal. e.g. A Service Level Agreement is not met, a customer loses business because of that, and sues us for the losses.
Data breach. e.g. There is an email with an invoice from customer "A" sent to customer "B", leaking GDPR protected information.
Economic loss. e.g. An expensive product was bought, but an error caused two of them to be sent.

Compromise Decision of Black Boxed Tasks

Once taken the concerns into account, a decision has to be made about handling possible failure for the task:

⚠️ This is about possible failure it is often possible to guarantee that a task has failed, if that’s the case, that failure will be treated as it would in a deterministic task.

At least once

If there’s a suspicion that the task has failed, it will be retried indefinitely, assuming the risk that it might happen more than once. It is important to understand that this means to attempt it until its success is registered, or it expires, meaning that it might even succeed indefinitely.

e.g. Blocking a user that has been deemed hostile should be attempted until success is registered.

Up to `n` times

If there’s a suspicion that the task has failed, it will be retried a certain amount of times, assuming the risk that it might happen more than once.

e.g. Sending a verification code as an SMS, it’s desirable to guarantee that it gets sent, but the costs could pile up rather quickly if the retries happen indefinitely for every user.

At most once

If there’s a suspicion that the task has failed, it’s given up silently.

e.g. Sending a welcome email.

Important: Human intervention required

If there’s a suspicion that the task has failed, there will be a step where a human has to make a decision. This is not a special kind of compromise, as there’s already a kind of task for this, the External Action Required.

e.g. A user requested the deletion of their private data, but the confirmation email could not be sent, it is desirable to reach out to the user to avoid legal risks.

The conclusion

The outcome of a task that has been deemed possibly failed needs to be mapped to a business outcome, that is, a transition. All the other conclusions are mapped just like the deterministic ones.

Algorithms for each kind of Task

Every task requires:

Task identifier

A unique, human-readable identifier that is unique for the action that is being attempted and that has to be consistently and deterministically generated only with the information contained in the event that triggered it. It will be referred to as the Task identifier. Optionally, they can be part of the signature of the events that trigger a task. Some examples would be:

refund-eaf1c4d0-3cfd-4fb4-bae0-864a6142446d
partial-payment-invoice-6ddcfdc3-8769-4f53-9cb8-66f6d6aa9b69-part-0 In this case, the event needs to carry the part, number and the aggregate responsible for it needs to keep track of which partial payment it is emitting the event for.
payment-initiated-6ddcfdc3-8769-4f53-9cb8-66f6d6aa9b69

Expiration functions

A function that receives the event that triggered the task, its Task identifier, and determines whether it still needs to be performed.
A function that maps the expiration to one of the possible transitions of the task.

To-Do Item

Tasks that expose an endpoint that a third party must complete, this might even be a human actor. The abstraction requires the following definitions:

A read model definition.
An endpoint definition:
1. Access control.
2. A data contract.
3. Type.
  1. Message queue.
  2. Synchronous API route segment (web API, GraphQL…)
4. A function that queries the read model based on the information in the data received.
5. A function that makes a decision based on both the read model and the data received, and nothing else.
A function that receives the event that leads to this task and generates a read model.

The algorithm:

State

Tasks that receive an event, attempt an action and, once successful, make a decision and emit a new event.

Retriability

All state tasks can define retry logic, this is done through a back off function that receives how many times the function has been attempted, the event, the Task identifierand an optional error type (an exception in most languages), and returns either an amount of time to wait until the next attempt or a decision to not try again.

To implement this function, it is important to take into account the “retriability” of a task based on the risk assessment made in the business concern section.

At least once
Up to n times
At most once

In general, tasks falling into the first category can be retried until the timeout function stops the cycle. If that is the case, no retry back-off function is needed.

Fault tolerance

All state tasks need to define a function that takes an error type, represented as exceptions in most languages, the event, and the Task identifier, and returns a business decision. This function will be called if the retriability function decides not to retry again.

Idempotency enabled

This task is an integration with a third party that provides a mechanism to uniquely identify an action, often with an idempotency header, so the action can be attempted many times without producing the effect several times. The definition requires:

A function that receives the event and the Task identifier, attempts the action and with the result of it, makes a decision.

The algorithm:

Optimistic lock enabled

This task is an integration with a third party that allows to perform an action in two steps, and determine whether the first step of the task is already underway by another process by querying for the first step based on some form of metadata that includes the Task identifier. The definition consists on:

A function that receives the event and the Task identifier, and attempts the first step of the task.
A function that receives the event and the Task identifier, and determines whether the action has already started.
A function that receives the event and the Task identifier, attempts the second step, and based
An optional function that receives the event and the Task identifier, and undoes any incomplete attempt of the task.

The Task will:

Attempt the first step.
Verify that the first step just made is the oldest one not expired for this identifier.
1. If it isn't, it tries to delete the step and exits without an outcome.
Verify that there isn't a completed second step for this identifier.
1. If there is, it tries to delete the step and exits without an outcome.
Attempt the second step.
Generate an outcome with the result of the second step.

The algorithm:

Pessimistic lock enabled

This task is an integration with a third party that allows to query if an attempt on the action has been performed before based on some form of metadata that includes the Task identifier. If it has, it has to be possible to get the outcome of that previous attempt. The definition consists on:

A function that receives the event and the Task identifier, attempts the task, and uses its outcome to make a decision.
A function that receives the event and the Task identifier, and determines whether the action has been tried before. If it has, reads the outcome of the task and makes a decision.
A property that delimits how long can the task take in an attempt, cancelling it after that time.

The Task will:

Write a lock for the Task identifier.
Verify that the lock just made is the oldest one not expired for this identifier.
1. If it isn't, aborts without an outcome.
Verify that there isn't a completed action in the service for this identifier.
1. If there is, it is guaranteed that the original process has expired, so emit an outcome with the result available.
Attempt the action.
Generate an outcome with the result.

The algorithm:

Black boxed

This task in an integration with a third party that offers no way to determine whether an action or an attempt has been performed before. While it is possible to determine whether an attempt has started, it is not possible to determine if the third party actually received a request from the system unless the third party replied and the system was successful in storing the outcome.

Since the task is attempted via a pessimistic lock, it is possible this means that if an expired lock exists and no decision has been recorded, the state of possible failure is reached.

The definition of this task consists on:

A function that receives the event and the Task identifier, attempts the task, and uses its outcome to make a decision.
A function that receives the event, the Task identifier, and the time the previous attempt started and makes a decision.

The Task will:

Write a lock for the Task identifier.
Verify that the lock just made is the oldest one.
1. If it isn't, verify that every previous lock is past expiration, and with a registered fault.
  1. If they all meet both conditions, continue.
  2. If there is at least one expired without a fault, emit an uncertainty outcome.
  3. If there is at least one that is neither expired nor faulted, register a fault for this attempt and abort without an outcome.
Attempt the action.
1. If there's a fault, register the fault.
Generate an outcome with the result.

The algorithm:

The Workflow

The core tenets of the framework

Rejection is not failure

The concept of a Task is, when it comes to its outcome not concerned about which outcome means a success business wise, but about whether the action could be executed and a decision was made out of the outcome. This means that any possible result just represents the next possible state.

Undo "A" is just do "B" when "A" has been done

While the Task is not concerned about business failure, the Workflow itself is, there are actions that might need to be undone if subsequent Tasks fail, this is not considered a special action in this framework, but just an action to perform after a certain precondition has been met.

All parties are third parties

Even when the services integrated are developed, maintained and owned by the same company, even by the same team, treating any external service as a third party that can fail to respond leads to predictable software. While it might seem extreme, bear in mind that this treatment is done only once, as part of the framework, so the pressure of having to consider every outcome is released once designing the state machine.

Yes, even people are third party services

In a To Do Item, an external action is awaited, this might come from an employee of the company owning the system, a confirmation from a user, or even a user from the third party needing to click on a link. The expiration scenario, that is, when the action never happens, must be a clearly visible part of the design, as it is not a matter of “if”, but of “when” will it happen.

Insist, desist or react

As complex as the decisions for faults in a Task might seem, there are only three possible outcomes:

Retry the action, this is part of the retriability definition of the Task.
Desist in the attempt, that is, deciding not to retry, and in the fault mapping function, continue the workflow towards a different State.
React to the fault, that is, deciding not to retry, and in the fault mapping function, continue the workflow towards a To Do Item.

Optimism through deterministic pessimism

Unless there's a performance problem actually happening, treat both optimistic and pessimistic integrations as a pessimistic one.

⚠️ Since an optimistic integration needs to happen in two steps, simply define every step as a different integration.

⚠️ Remember that a performance concern is not a performance problem actually happening, the robustness of pessimistic locked actions allow for an aggressive distribution of the tasks amongst workers, reducing the time required to execute thousands of tasks to the longest execution time of a single one.

The framework, by example

The following example is the creation of an order. So simplify the model, any fault and expiration result that maps to a default exit, Order cancelled in this case, can be left as an implicit mapping.

Legend

Glossary

Code safety:

Property of code that handles every possible state it can go through. A safe function has constraints in its input, so every possible value that is allowed by the compiler is valid, never throws exceptions for any valid value of its input, and always returns a valid value, specified by its return type.

Completed task:

A task that has executed without fault. This definition is not concerned whether the action was accepted or rejected.

e.g. A fraud check that returns a result, regardless of the user being deemed trustworthy or not, has completed.

Concurrency handling:

Capacity of a system to behave predictably when interacting with a resource that can be accessed by another process, be it a third party, another instance of the current system or another process that is part of the same system.

Failed task:

A task that could not be completed after all the fault tolerance mechanisms have exhausted any option to recover, or the task can’t be performed for other reasons.

e.g. An email provider throttled a request, a circuit breaker acts in to give the service some time to recover, after the request successfully goes through, it returns a Bad Request response because the content had some invalid characters.

e.g. A SMS provider rejects an attempt to send a message because the associated account does not have enough credits.

Fault tolerance:

Capacity of a system to continue functioning predictably after an external action fails to execute as expected, it is assumed that the fault is transient, that is, that the external system will be able to eventually respond.

Optimistic lock enabled action:

An integration that can be done in two steps, provides a deterministic mechanism to identify the process as unique, and allows for the process to be cancelled.

Pessimistic lock enabled action:

An integration that provides a deterministic mechanism to identify the process as unique, but lacks the other two characteristics of the optimistic lock enabled

Coaches don’t play

Rubén Domínguez Falcón — Wed, 02 Nov 2022 09:27:11 +0000

I remember the first time I was promoted to a leadership position. I was thrilled, looking forward to all the new challenges this new position would bring. But I also remember the week after, how I found myself trying to complete the same tasks as before, but now also juggling additional meetings and management tasks.

Clearly, something was wrong; becoming a leader shouldn't be "doing the same work but having more meetings".

And it’s definitely not. I was seeing it from the wrong angle. I was about to learn that being a leader means you are not a player anymore.

You are not a player anymore.

Imagine for a second that you are a soccer player and you're promoted to be the team's coach. Can you imagine yourself on the pitch, trying to score during the match?

Yeah, it sounds ridiculous! Coaches are not supposed to play the match, but to guide the team. The same applies to any leader. Your goal is to guide your team to achieve the best results, not to do their job.

As a leader, you should focus on defining the strategy, clarifying goals, setting the right expectations, providing your team with the tools to succeed, and then letting them play the match.

You should interiorize this concept because you'll find a lot of challenges that will test your leadership skills, like the delivery trap.

The delivery trap

All projects have milestones and deadlines; the closer the deadline, the more conscious you become about the performance gaps in your team. In these situations, you might feel tempted to jump in and start doing their work.

You'll find a lot of good reasons to jump in, "I want to lead by example", "It will be faster if I do it myself", "I want to support my team"...

They are all excellent reasons, but it’s also the wrong decision. Everybody loses if you jump in and start working on your team's tasks.

By jumping in, you are:

Removing the opportunity for your team to face the challenge and learn from it.
Losing the opportunity to improve your leadership skills, because you’ll only be polishing the skills you already had from your old position.

You should focus on improving your leadership skills to help your team.
With the pressure of a deadline, you should focus on redefining the plan, ensuring the expectations are clear, figuring out what's blocking your team, and helping them to perform at their very best.

That way, iteration by iteration, your team will become better. Your ultimate goal should be to make your team grow to the point that they don't need you. The goal is to make yourself redundant.

Make yourself redundant

One of the many good things about being in a leadership position is that it allows you to help people grow and develop their potential. If you do it well, after some time, your team will reach a point where they will be able to do their job without needing you; that's the ideal scenario.

Once your team doesn't need you, you'll have the opportunity to join a different team and continue helping there. That is one of your key goals as a leader, to help people grow, and in doing so, help the company grow.

Swiss Army knives or Allen keys?

Sebastián Pérez Pérez — Tue, 27 Sep 2022 09:58:07 +0000

When you have to choose a tool to perform a task, you always have two options. You can go to the shop and buy a super versatile Swiss Army knife that will provide you with a lot of options for many different tasks, or you can go and buy a specific tool for a specific task. It will work perfectly for what you need it for, but you won’t be able to use it for anything else, like an Allen key.

With the Swiss Army knife, you’ll be able to tighten certain screws, cut a piece of string, break a branch, build a spear, etc. On the other hand, with the Allen keys, you will be able to tighten hex screws only. So, it looks like an easy decision, doesn’t it? Go and buy the Swiss Army knife and you’ll have a tool that you can use for many situations, and you know it won’t just be thrown into your toolbox and never used again…

But … what if you need to tighten loads of hex screws every day and have to do it quickly? Now, the decision to buy the Swiss Army knife doesn’t sound as obvious. Even though you know that you’ll use the Allen key just for that specific task, you’d probably prefer to have the best tool for the job.

In development, when we have to build a team and distribute the work between the team members, we have to make a similar decision. Do we choose the Swiss Army knife or the Allen key?

If I told you that you could have a team where all the team members could develop, test, define tasks, create the infrastructure, manage the deployments, automatize CI/CD pipelines, etc, and do it all really well and fast, it would sound like a dream come true, and you would sign up without question. But achieving that is no easy feat. It would take a long time and a lot of effort from the whole team, and that investment would also have to be applied to new members that recently join the team.

So if you think of the long term, and assuming that time is not a concern, I would say “Go for it! You will build the best team ever”. But working on a project where the timeline is not a constant concern doesn't usually happen, because in our business, time is money.

For most people however, the normal situation is to have strict deadlines, because the client is waiting for an application, and the team has committed to delivering the product by a specific date. Usually, whatever that timeframe is, it’s never enough and we always have a lot to do during the final days leading up to the deployment.

So, that means we need to optimize resources, and try to get the best out of everyone if we want to meet the deadline successfully. And in that kind of situation where we have adjusted timelines, having specialists for each task would be the best option. We can take advantage of each person’s strengths, and as a team, achieve our objective.

This is the eternal debate of “specialization vs versatility” but history tells us with Taylorism, Fordism and Toyotism, that when productivity and efficiency become key for any requirement, specialization is the shorter and more successful path. And I have to say that I agree with history. Having a team where you could put anyone on to any task without affecting the performance of the team is simply not realistic. Everyone has their own strengths and weaknesses, and everyone has their own skill set. Having a team of clones, where everyone is a champion at everything only happens in the best sci-fi movies. In my opinion, a good team leader should know their team and take advantage of the capabilities of each person. They should be able to configure the team where they complement each other, leveraging their strengths and focusing on the tasks that they can nail.

Of course, investing in learning should be a top priority for any company, because at the end of the day people are what really matter. They must be the center around which the company should be built. And the only way to achieve that is to provide time and create an environment where your team can learn, increase their skills and grow professionally. They are the seeds, and with the right environment, they will grow into an extremely strong team. That is the long term goal, a team of Swiss Army knives. But when a project requires fast and decisive decisions and actions, choosing specialization over versatility is the key.

Web Pentesting Learning - Beginner edition

J2RGEZ — Thu, 01 Sep 2022 09:37:07 +0000

After a couple of years of learning on my own, I created a brief list of the assets I think were most useful for me at the time of learning web pentesting. Hope you find it helpful!

Books

Linux Basics for Hackers: Getting Started with Networking, Scripting, and Security in Kali, by OccupyTheWeb. => Beginner friendly and very well written.
Penetration Testing: A Hands-On Introduction to Hacking, by Georgia Weidman => In my opinion it’s a bit outdated and some parts are difficult to understand but still a very good book.
Web Hacking 101, by Peter Yaworski => A summary of all common web vulnerabilities with examples.
Hands-On Penetration Testing on Windows, by Phil Bramwell => Also focusing on the registry tree and how the most common keys work (such as kerberos keys that handles authentication). Microsoft official docs are good for this.

Twitter accounts

@three_cube a.k.a OccupytheWeb. This is from the author of the first book listed above and also has a very good hacking blog: https://www.hackers-arise.com/
@theXSSrat
@stokfredik =>The coolest hacker ever! He also has a great YouTube channel
@NahamSec
@TheHackersNews
@thecybermentor and @TCMSecurity

Overall #infosec twitter is a very good place to start reading writeups and latest news.

Platforms to get your hands dirty

If you’re just starting, I recommend Tryhackme. It’s amazing! And it has a lot of walkthrough boxes (just be aware of this). Then, I would jump to Hackthebox which has the most realistic machines. In my opinion pentesterlab is a bit expensive for the quality their competitors have for almost half the price, but their certificates are good.

Bug bounty platforms

Bug bounty is about hacking as a freelancer, but it’s nice to read writeups (if public), as these are real business-level vulnerabilities. Also reading about bug bounty will teach you tricks to increase your speed and overall organization, which is one of the key skills you need in this category. Why? Because you want to report your findings before anyone else to get paid and avoid duplicates. Hackerone is one of these platforms: https://hackerone.com/directory/programs?order_direction=DESC&order_field=resolved_report_count

Tools you should know (basic level)

Metasploit
sqlmap
nmap
Burp Suite or any other web scanner alternative
Hydra login cracker
Any hash cracker like john the ripper, hashcat, etc
Shodan: at least to understand what it is, in case you want to use it one day

FAQ

What are writeups? Instructions about how someone hacked something in detail.

I’m most probably missing something here, but I think it’s a nice starting point. Let me know in the comments if you would add anything else to this list and/or what helped you when you started learning pentesting.

How to build a native iOS dApp that uses the Flow blockchain as the backend

Juan Sagasti — Thu, 07 Jul 2022 22:59:11 +0000

Introduction

This is the second and final part of a web3 tutorial about the Flow blockchain. Understanding the first part is essential and a pre-requisite because I explain there how to create and test Smart Contracts, the one we developed to create notes, its API, and how to write transactions and scripts to mutate and read data in the blockchain. Please take your time and make sure you understand it before continuing :)

First part: A notepad in the Flow blockchain

So, in this final part, you will learn:

How to deploy smart contracts in a Flow blockchain network: testnet.
How to build an iOS dApp that communicates with that contract in Flow's testnet. For user authentication, the app will use the Blocto Wallet, a cross-chain crypto wallet compatible with Flow. I assume you have a basic understanding of iOS programming. We will be using SwiftUI + Combine.

Deploying the Smart Contract

Note that the NotepadManager contract we will be using in the iOS dApp is already deployed in a flow account of my own. Remember that you can use any public smart contract in the blockchain, so you don't need to deploy the contract again in every account that wants to use it, as we learned in the previous article. This step is only for learning purposes in case you are interested in how to deploy contracts. You can skip this section and go directly to the dApp implementation part.

We are going to deploy the NotepadManager contract to the testnet, which is nearly identical to the mainnet, but we can operate without paying real gas. This is ideal for learning and testing our projects before deploying them to the production network (mainnet).

Install the Flow CLI
Generate the account keys. Remember to safely store the private key and don't share it with anyone.
Create a Testnet flow account here with your public key.
Create a folder for your contracts project, open it in the terminal, and do flow init. This will create a flow.json file with a default configuration.
Open your project folder in an editor. I'm using Visual Studio Code, which has an excellent Cadence plugin you can install from the Extensions Marketplace.
Add a new file to your project called NotepadManager.cdc. Paste the contract code we developed in the previous article. You can find it in the Playground, too.
Edit the flow.json file so it looks like this:

{
    "emulators": {},
    "contracts": {
        "NotepadManagerV1": "NotepadManagerV1.cdc"
    },
    "networks": {
        "emulator": "127.0.0.1:3569",
        "mainnet": "access.mainnet.nodes.onflow.org:9000",
        "testnet": "access.devnet.nodes.onflow.org:9000"
    },
    "accounts": {
        "my-testnet-account": {
            "address": "YOUR ACCOUNT ADDRESS (without the '0x' prefix)",
            "key": "YOUR PRIVATE KEY"
        }
    },
    "deployments": {
        "testnet": {
            "my-testnet-account": ["NotepadManagerV1"]
        }
    }
}

Note that we emptied the "emulators" field because we won't be using the local emulator but a real network. We also added the "contracts" and "deployments" fields to point to our contract and account.

Remember to fill the "address" and "key" fields, as specified. You obtained your account address in step #3 and the private key in step #2. Be careful with that key, and don't commit it to a public repository.

Now that everything is set up, it's time to deploy the contract! In the terminal, in your project folder, do:

flow project deploy

You'll see something like this:

  Deploying 1 contracts for accounts: my-testnet-account
  NotepadManagerV1 -> 0x9bde7238c9c39e97 (09d8818a9d41ae2fc92f9dd6d7355772e7d97e28fb9e20b2f66f473420a16b31)

Which can be read as:

NotepadManagerV1 -> the address containing it (the transaction ID)

You can see the transaction status or the account details through blockchain explorers like Tesnet Flowscan or Flow View Source. For example, if I introduce my account address 0x9bde7238c9c39e97 in Flowscan:

You can see the account balance, transaction history, and contracts deployed to the account. If you click the NotepadManagerV1 link, you can see the contract's code! This is powerful for public audits of the contracts in the blockchain.

Implementation

Our iOS dApp will have a pretty straightforward architecture:

All the elements in that picture are .swift files. You can start by creating an Xcode project (iOS 15.5 for the time being) and all those files. Files with the ...View suffix are SwiftUI views.

For communicating with the Flow blockchain, we will be using the iOS SDK that Outblock is building. A big update containing breaking changes is planned to be released soon, so we are going to lock now the SDK version to a particular commit (latest as of today) to be sure our code keeps working overtime. To do that, just add the dependency of the Swift Package Manager tied to the b4a9cfaaa0003f3f250490e6ccdf8b75065199a3 commit:

Now we can start adding code to our Swift files!

`FlowNotesApp.swift`

This is the entry point of our app. It's in charge of initializing the fcl-swift framework with the needed endpoints to communicate with the Flow blockchain and Blocto Wallet and showing the LoginView or the MyNotesView depending on the user's authorization state.

import Combine
import FCL
import SwiftUI

enum UI {
    static let flowGreen = Color(red: 0, green: 1, blue: 130/255)
}

@main
struct FlowNotesApp: App {
    @State private var currentUser: User?

    init() {
        // FCL configuration with the Tesnet Nodes URLs needed 
        // to stablish the connection with the blockchain and 
        // the Blocto Wallet:
        fcl.config(appName: "FlowNotes",
                   appIcon: "https://placekitten.com/g/200/200",
                   location: "https://foo.com",
                   walletNode: "https://fcl-discovery.onflow.org/testnet/authn",
                   accessNode: "https://access-testnet.onflow.org",
                   env: "testnet",
                   scope: "email",
                   authn: "https://flow-wallet-testnet.blocto.app/api/flow/authn")

        let flowGreenColor = UIColor(cgColor: UI.flowGreen.cgColor ?? UIColor.green.cgColor) // Color to UIColor conversion
        UINavigationBar.appearance().largeTitleTextAttributes = [.foregroundColor: flowGreenColor]
        UINavigationBar.appearance().titleTextAttributes = [.foregroundColor: flowGreenColor]
    }

    var body: some Scene {
        WindowGroup {
            VStack {
                if currentUser != nil {
                    MyNotesView(vm: MyNotesVM())
                } else {
                    LoginView()
                }
            }
            .preferredColorScheme(.dark)
            .onReceive(fcl.$currentUser) { user in
                currentUser = user
            }
        }
    }
}

`LoginView.swift`

Simple screen with a big login button that invokes the authenticate() method on the LoginVM ViewModel.

`LoginVM.swift`

The authenticate() method will invoke the FCL's authentication method, which will present us with the Blocto Wallet webView for the user to log in. After successful authentication, the FCL's currentUser property will be populated, triggering an update in our FlowNotesApp class, which is listening to that property. The MyNotesView will be shown then.

import Combine
import FCL
import Foundation

final class LoginVM: ObservableObject {

    private var cancellables = Set<AnyCancellable>()

    func authenticate() {
        fcl
            .authenticate()
            .receive(on: DispatchQueue.main)
            .sink { result in
                print(result)
            } receiveValue: { response in
                print(response)
            }
            .store(in: &cancellables)
    }
}

`MyNotesView.swift`

This view is in charge of showing the list of notes retrieved by MyNotesVM. The users can also:

Delete notes with the swipe-to-left gesture.
Present the NoteEditorView from the "+" button to create new notes.
Show the action menu from the "..." button. Here, users can manually refresh the list, delete the entire Notepad or sign out.

You will note that the transactions are pretty slow compared to a regular backend. This is because mutations of the blockchain need to be accepted and mined, which could take several seconds to happen. For simplicity, we present a full-screen loading screen while the VM's state is loading, until the transaction is sealed and a script can safely retrieve the final result. Read-only operations (scripts) like the notes fetch are blazing fast. If the transaction fails with an error, we present it to the user with an AlertView.

As you see, the transactions and script code used here are nearly identical to the ones in the first article (I omitted the comments in these to avoid repetition):

import SwiftUI

struct MyNotesView: View {

    @StateObject private var vm: MyNotesVM

    @State private var isLoading = false
    @State private var isShowingNoteEditor = false
    @State private var isShowingErrorAlert = false

    init(vm: MyNotesVM) {
        _vm = .init(wrappedValue: vm)
    }

    private var signOutButton: some View {
        Button {
            vm.signOut()
        } label: {
            Label("Sign Out", systemImage: "person.crop.circle.badge.xmark")
        }
    }

    private var refreshNotepadButton: some View {
        Button {
            vm.queryNotes()
        } label: {
            Label("Refresh notepad", systemImage: "arrow.clockwise")
        }
    }

    private var deleteNotepadButton: some View {
        Button {
            vm.deleteNotepad()
        } label: {
            Label("Delete notepad", systemImage: "trash")
        }
        .opacity(vm.notes != nil ? 1 : 0)
        .disabled(vm.notes == nil)
    }

    private var moreOptionsButton: some View {
        Menu {
            refreshNotepadButton
            Divider()
            deleteNotepadButton
            Divider()
            signOutButton
        } label: {
            Image(systemName: "ellipsis")
        }
    }

    private var addNoteButton: some View {
        Button {
            isShowingNoteEditor = true
        } label: {
            Image(systemName: "plus")
        }
    }

    var body: some View {
        ZStack {
            NavigationView {
                Group {
                    if let notes = vm.notes {
                        List {
                            ForEach(notes, id: \.id) { note in
                                NoteRow(note: note)
                                    .listRowBackground(Color.black)
                            }
                            .onDelete { index in
                                vm.deleteNote(atIndex: index.first)
                            }
                        }

                    } else if vm.state == .loading {
                        EmptyView()
                    } else {
                        Text("Your notepad is empty 🤓\nAdd a new note from\nthe'＋'button")
                            .multilineTextAlignment(.center)
                            .font(.headline)
                            .foregroundColor(.gray)
                    }
                }
                .navigationBarItems(leading: moreOptionsButton, trailing: addNoteButton)
                .navigationTitle("Flow Notes")
            }
            .tint(UI.flowGreen)

            ZStack {
                Color.black
                    .edgesIgnoringSafeArea(.all)
                    .opacity(isLoading ? 0.7 : 0)

                ProgressView {
                    Text("On it! 🚀 \nPlease wait...")
                        .multilineTextAlignment(.center)
                        .foregroundColor(.white)
                }
                .progressViewStyle(CircularProgressViewStyle(tint: .white))
                .opacity(isLoading ? 1 : 0)
            }
        }
        .onChange(of: vm.state) { newValue in
            isLoading = newValue == .loading
            isShowingErrorAlert = newValue.isFailed
        }
        .alert(isPresented: $isShowingErrorAlert, content: {
            Alert(title: Text("Error!"),
                  message: Text(vm.state.errorMessage ?? "Unknown"),
                  dismissButton: .default(Text("OK"), action: {
                vm.currentErrorDidDismiss()
            }))
        })
        .sheet(isPresented: $isShowingNoteEditor) {
            isShowingNoteEditor = false
        } content: {
            NoteEditorView().environmentObject(vm)
        }
    }
}

struct NoteRow: View {
    let note: Note

    var body: some View {
        HStack {
            VStack(alignment: .leading, spacing: 5) {
                Text(note.title)
                    .foregroundColor(.black)
                    .font(.body.weight(.bold))

                Text(note.body)
                    .foregroundColor(.black)
                    .font(.caption)
            }

            Spacer()
        }
        .padding()
        .background(UI.flowGreen)
        .cornerRadius(10)
    }
}

private struct PreviewWrapper: View {

    var vm = MyNotesVM()

    init() {
        vm.notes = [Note(id: 0, title: "This is a great note", body: "This is the body of a great note."),
                    Note(id: 1, title: "Another note", body: "The best body that was ever written.")]
    }

    var body: some View {
        MyNotesView(vm: vm).preferredColorScheme(.dark)
    }
}

struct MyNotesView_Previews: PreviewProvider {
    static var previews: some View {
        PreviewWrapper()
    }
}

`MyNotesVM`

This is the VM that defines the set of operations available to mutate the notepad through transactions or fetch the notes through a script.

Every time you send a transaction, the SDK will present a transaction confirmation screen (webView) from the Blocto Wallet, so you can review the transaction's code and approve it:


import Combine
import FCL
import Flow
import Foundation

enum LoadingState: Equatable {
    case idle
    case loading
    case didFail(error: String)

    var isFailed: Bool {
        guard case .didFail = self else { return false }
        return true
    }

    var errorMessage: String? {
        guard case let .didFail(error) = self else { return nil }
        return error
    }
}

final class MyNotesVM: ObservableObject {

    @Published private(set) var state = LoadingState.idle
    @Published var notes: [Note]?

    private let notepadManagerAddress = "0x9bde7238c9c39e97"
    private var cancellables = Set<AnyCancellable>()

    init() {
        defer {
            queryNotes()
        }
    }

    // MARK: - Public API

    func createNote(title: String, body: String) {
        guard fcl.currentUser?.addr != nil else { return }

        state = .loading

        fcl.mutate {
            cadence {
                // Transaction that checks if the Notepad 
                // exists (and creates it if needed) before 
                // creating and adding the note to it:
                 """
                 import NotepadManagerV1 from \(notepadManagerAddress)

                 transaction {
                     prepare(acct: AuthAccount) {
                         var notepad = acct.borrow<&NotepadManagerV1.Notepad>(from: /storage/NotepadV1)

                         if notepad == nil { 
                             // Create it and make it public
                             acct.save(<- NotepadManagerV1.createNotepad(), to: /storage/NotepadV1)
                             acct.link<&NotepadManagerV1.Notepad>(/public/PublicNotepadV1, target: /storage/NotepadV1)
                         }

                         var theNotepad = acct.borrow<&NotepadManagerV1.Notepad>(from: /storage/NotepadV1)
                         theNotepad?.addNote(title: "\(title)", body: "\(body)")
                     }
                 }
                 """
            }

            gasLimit {
                1000
            }
        }
        .receive(on: DispatchQueue.main)
        .sink { completion in
            if case let .failure(error) = completion {
                self.state = .didFail(error: error.localizedDescription)
            }
        } receiveValue: { [weak self] transactionId in
            guard let self = self else { return }

            self.waitForSealedTransaction(transactionId) {
                self.queryNotes()
            }
        }
        .store(in: &cancellables)
    }

    func deleteNote(atIndex index: Int?) {
        guard fcl.currentUser?.addr != nil, let index = index, let idToDelete = notes?[index].id else { return }

        state = .loading

        fcl.mutate {
            cadence {
                // Transaction that tries to delete the note 
                // if the Notepad exists:
                 """
                 import NotepadManagerV1 from \(notepadManagerAddress)

                 transaction {
                     prepare(acct: AuthAccount) {
                            let notepad = acct.borrow<&NotepadManagerV1.Notepad>(from: /storage/NotepadV1)
                            notepad?.deleteNote(noteID: \(idToDelete))
                     }
                 }
                 """
            }

            gasLimit {
                1000
            }
        }
        .receive(on: DispatchQueue.main)
        .sink { completion in
            if case let .failure(error) = completion {
                self.state = .didFail(error: error.localizedDescription)
            }
        } receiveValue: { [weak self] transactionId in
            guard let self = self else { return }

            self.waitForSealedTransaction(transactionId) {
                self.queryNotes()
            }
        }
        .store(in: &cancellables)
    }

    func deleteNotepad() {
        guard fcl.currentUser?.addr != nil else { return }

        state = .loading

        fcl.mutate {
            cadence {
                // Transaction that deletes the Notepad:
                 """
                 import NotepadManagerV1 from \(notepadManagerAddress)

                 transaction {
                     prepare(acct: AuthAccount) {
                         var notepad <- acct.load<@NotepadManagerV1.Notepad>(from: /storage/NotepadV1)!
                         NotepadManagerV1.deleteNotepad(notepad: <- notepad)
                     }
                 }
                 """
            }

            gasLimit {
                1000
            }
        }
        .receive(on: DispatchQueue.main)
        .sink { completion in
            if case let .failure(error) = completion {
                self.state = .didFail(error: error.localizedDescription)
            }
        } receiveValue: { [weak self] transactionId in
            guard let self = self else { return }

            self.waitForSealedTransaction(transactionId) {
                self.notes = nil
                self.state = .idle
            }
        }
        .store(in: &cancellables)
    }

    func queryNotes() {
        guard let currentUserAddress = fcl.currentUser?.addr else { return }

        state = .loading

        fcl.query {
            cadence {
                // Script that tries to get all notes from the 
                // current Notepad. It returns nil in case the 
                // Notepad doesn't exist yet publicly:
                """
                import NotepadManagerV1 from \(notepadManagerAddress)

                pub fun main(): [NotepadManagerV1.NoteDTO]? {
                    let notepadAccount = getAccount(0x\(currentUserAddress))

                    let notepadCapability = notepadAccount.getCapability<&NotepadManagerV1.Notepad>(/public/PublicNotepadV1)
                    let notepadReference = notepadCapability.borrow()

                    return notepadReference == nil ? nil : notepadReference?.allNotes()
                }
                """
            }
        }
        .receive(on: DispatchQueue.main)
        .sink { [weak self] completion in
            guard let self = self else { return }

            if case let .failure(error) = completion {
                self.state = .didFail(error: error.localizedDescription)
            }
        } receiveValue: { [weak self] result in
            print(result)

            guard let self = self else { return }
            guard let valuesArray = result.fields?.value.toOptional()?.value.toArray() else {
                self.notes = nil
                self.state = .idle
                return
            }

            let notes: [Note] = valuesArray.compactMap {
                guard let noteData = $0.value.toStruct()?.fields else { return nil }
                // It's not the best decoding code, but enough 
                // for what we are illustrating. The SDK 
                // maintainer is working on a better data 
                // decoder for future versions.
                let id = noteData[0].value.value.toUInt64()
                let title = noteData[1].value.value.toString()
                let body = noteData[2].value.value.toString()

                guard let id = id, let title = title, let body = body else { return nil }
                return Note(id: id, title: title, body: body)
            }

            self.notes = notes
            self.state = .idle
        }.store(in: &cancellables)
    }

    func currentErrorDidDismiss() {
        state = .idle
    }

    func signOut() {
        fcl.currentUser = nil
    }

    // MARK: - Private

    private func waitForSealedTransaction(_ transactionId: String, onSuccess: @escaping () -> Void) {
        // We are using a background thread because we don't 
        // want the .wait() to block the main event loop:
        DispatchQueue.global(qos: .userInitiated).async {
            do {
                // We want to wait for the transaction to be 
                // sealed (last transaction state) because we 
                // need the final result to be available in 
                // the blockchain. It's slower, but we don't 
                // want intermediate transaction states where 
                // you can still get empty errors or data when 
                // querying the blockchain with a Script.
                let result = try Flow.ID(hex: transactionId).onceSealed().wait()

                // And we update the state again in the main thread:
                DispatchQueue.main.async {
                    print(result)

                    if !result.errorMessage.isEmpty {
                        self.state = .didFail(error: result.errorMessage)
                    } else {
                        onSuccess()
                    }
                }
            } catch {
                DispatchQueue.main.async {
                    self.state = .didFail(error: error.localizedDescription)
                }
            }
        }
    }
}

`NoteEditorView.swift`

The NoteEditorView is just a form to compose and send a new note.

import SwiftUI

struct NoteEditorView: View {

    @EnvironmentObject private var vm: MyNotesVM
    @Environment(\.dismiss) private var dismiss

    @State private var noteTitle = ""
    @State private var noteBody = ""

    private var canSaveNote: Bool { !noteTitle.isEmpty && !noteBody.isEmpty}

    private var saveButton: some View {
        Button("Save") {
            vm.createNote(title: noteTitle, body: noteBody)
            dismiss()
        }
        .opacity(canSaveNote ? 1 : 0.5)
        .disabled(!canSaveNote)
    }

    var body: some View {
        NavigationView {
            VStack {
                VStack {
                    TextField("Title", text: $noteTitle)
                        .foregroundColor(.black)
                        .font(.body.weight(.bold))
                        .accentColor(.black)
                        .padding(.vertical, 7)

                    TextField("Body", text: $noteBody)
                        .foregroundColor(.black)
                        .accentColor(.black)
                        .padding(.vertical, 7)
                }
                .padding(.horizontal)
                .background(UI.flowGreen.opacity(0.5))
                .cornerRadius(10)

                Spacer()
            }
            .padding()
            .navigationTitle("New note")
            .navigationBarTitleDisplayMode(.inline)
            .navigationBarItems(trailing: saveButton)
        }
        .tint(UI.flowGreen)
    }
}

struct NoteEditor_Previews: PreviewProvider {
    static var previews: some View {
        NoteEditorView()
            .preferredColorScheme(.dark)
            .environmentObject(MyNotesVM())
    }
}

`Note.swift`

Just the Note model:

import Flow
import Foundation

struct Note {
    let id: UInt64
    let title: String
    let body: String
}

And that's it! I hope you find this helpful and that it inspires you to find more cool use cases. If you want to see a full demo video in action and check this project's source code, please refer to the FlowNotes repository.

There's an edit transaction use case in the NotepadManager contract that I haven't used in the app. I leave it to you as an exercise if you are motivated enough :)

Thank you for reading!

Web3 tutorial: A notepad in the Flow blockchain

Juan Sagasti — Tue, 28 Jun 2022 16:04:00 +0000

Introduction

While learning about web3 and blockchain technologies at The Agile Monkeys, we found out Flow. According to them, "Flow was designed for exceptional consumer and developer UX". And after playing with it for some time, I agree with that. Their developer UX is among the best, if not the best, we've tried. Some things that make Flow different are:

A greener web3 network. Flow starts with HotStuff, a proven Proof of Stake consensus mechanism, and adds a unique multi-node architecture to drive dramatic improvements in speed, throughput, and environmental friendliness without sharding or "layer two". This means Flow is the greenest web3 network among leading platforms.

Resource-oriented programming with Cadence, their own programming language. If you are familiar with Swift, Rust, or TypeScript, you will love Cadence and will be writing Smart Contracts in no time.
Playgrounds. An online editor to write and test your Smart Contracts without needing to deploy them to the main or test networks. You can focus first on the business logic. This is what we will be using in this tutorial.
Upgradeable Smart Contracts. Securely and transparently patch bugs and upgrade pre-specified parts of a smart contract.
No Sharding. On Flow, every application can be a platform. Smart contracts and user accounts on Flow can always interact with each other in one atomic, consistent, isolated, and durable (ACID) transaction. In other words: all applications on Flow can run in the same shared execution state. This ensures Flow apps benefit from great user experience and full composability, letting developers easily build on each other’s work. Sharding and layer 2 solutions break composability and reduce network effects for dApps and smart contracts by removing ACID guarantees from the execution environment.
Client libraries to help you build your dApps, Smart User Accounts, and more!

Implementation

First of all, why will we build a notepad on the blockchain? That makes no sense! I know paying gas fees for creating notes wouldn't be an ideal use case. But the Internet is already crowded with NFTs tutorials, and I wanted a more classic approach for non-web3 devs that illustrates how Smart Contracts + the user account in Flow can act as the API and the backend of our hypothetical dApp. And hopefully, this will give you more ideas about other possible dApps!

Cadence is a resource-oriented programming language. This means it makes it easy to work with digital assets (resources) as if they were physical assets. A resource can only be in one place at a time. So, for example, if you assign (this is, move with the <- operator) a resource to a new variable, the resource won't be available in the original variable anymore, and that variable will be invalid:

let movedResource <- originalResource

This makes the experience of working with digital assets pretty natural. A resource can't be left forgotten in a variable. It must live in your user account or should be destroyed at the end of the function you are currently executing. Otherwise, the compiler will complain about the resource loss. It's pretty safe in this regard.

I encourage you to read the quick Introduction to Cadence and the Cadence Language Reference before continuing this tutorial if you want to understand things better. Much of the language's syntax is inspired by Swift, Kotlin, and TypeScript, so you could feel at home if you are used to them.

Open a Flow Playground; you will see something like this:

You'll find the test accounts, the transactions, and the script folders in the left sidebar. Transactions are mutations of the blockchain and require gas fees. Scripts are read-only, blazing-fast operations that allow you to perform queries without paying gas fees. Anyway, we don't need to worry about transaction fees in this tutorial.

The user's account filesystem domain structure can be seen as this:

According to Flow docs:

The storage domain is where all objects (such as structs and resource objects representing tokens or NFTs) are stored. It is only directly accessible by the owner of the account.

The `private domain is like a private API. You can optionally create links to any of your stored assets here. Only the owner and anyone they give access to can use these interfaces to call functions defined in their stored assets.

The public domain is your account's public API. It is like the private domain in that the owner can link capabilities here that point to stored assets. The difference is that anyone in the network can access the public domain's functionality. A Transaction in Flow is defined as an arbitrary-sized block of Cadence code that is signed by one or more accounts.

Transactions have access to the /storage/ and /private/ domains of the accounts that signed the transaction. Transactions can also read and call functions in public contracts and access public domains in other users' accounts.

I wanted to deploy a public contract in one user address and let the other users use it without needing to deploy the contract to every account that wants to use it. So let's start by selecting the 0x01 user account in the sidebar, removing the example code, and pasting our contract code. The code includes comments so you can understand the reasoning behind every part as part of this tutorial:
`

// There are different access levels, as defined in 
// https://docs.onflow.org/cadence/language/access-control/#gatsby-focus-wrapper, 
// but we want our contract to be accessible from outside this 
// account, so we defined it as public (pub). 
pub contract NotepadManagerV1 {
    // This var is just for fun, to know how many Notepads 
    // this contract created.
    pub var numberOfNotepadsCreated: UInt64

    // The Notepad resource. It contains the notes as nested 
    // resources in the 'notes' dictionary. 
    // It also defines the Notepad API.
    pub resource Notepad {

        // Dictionary of notes. We will use the note.uuid 
        // (UInt64) as the indexing key:
        pub var notes: @{UInt64 : Note}

        init() {
            // When a Notepad is created, we initialize the 
            // 'notes' property with an empty dictionary:
            self.notes <- {}
        }

        // Every Resource needs a 'destroy' method.
        destroy() {
            // As we have nested resources, we need to destroy 
            // them here too:
            destroy self.notes
        }

        // Public method to add a note to the user's Notepad.
        pub fun addNote(title: String, body: String) {
            // Create a new note Resource and move it to the 
            // constant:
            let note <- create Note(title: title, body: body)

            // As we are using a dictionary of resources and 
            // we are adding a new note with the note.uuid as 
            // the indexing key, Cadence requires you to 
            // handle the note that could already be in that 
            // dictionary's position to avoid a Resource loss.
            // So we move what could already be in that 
            // position to the 'oldNote' constant and destroy 
            // it. In this case, that won't do anything
            // because 'oldNote' will be nil (we are adding a 
            // new note to that position with a new UUID). 
            // And we move the new note to the new position. 
            // This chained move (<-) operators instruction 
            // could be read as: "We move the new note to that 
            // position, and what was previously in that 
            // position to a constant, so we don't lose it":
            let oldNote <- self.notes[note.uuid] <- note
            destroy oldNote
        }

        // Public method to edit a note in the Notepad.
        pub fun editNote(noteID: UInt64, newTitle: String, newBody: String) {
            // We create and move a new note to the previous 
            // note position, and the old note to the 
            // 'oldNote' constant (and then destroy it):
            let oldNote <- self.notes.insert(key: noteID, <- create Note(title: newTitle, body: newBody))
            destroy oldNote
        }

        // Public method to delete a note from the Notepad.
        pub fun deleteNote(noteID: UInt64) {
            // Move the desired note out from the dictionary 
            // (to a constant) and destroy it: 
            let note <- self.notes.remove(key: noteID)
            destroy note
        }

        // Public function that, given a note resource id in 
        // the Notepad, returns a NoteDTO from it (see this 
        // DTO definition below). 
        // This will be useful for client queries (Scripts), 
        // which are read-only, and we cannot just return a 
        // Resource there. 
        pub fun note(noteID: UInt64): NoteDTO {
            // We take the note out of the notes dictionary, 
            // create a NoteDTO from it, put the note Resource 
            // back into the dictionary, and return the NoteDTO:
            var note <- self.notes.remove(key: noteID)!
            let noteDTO = NoteDTO(id: noteID, title: note.title, body: note.body)

            let oldNote <- self.notes[noteID] <- note
            destroy oldNote
            return noteDTO
        }

        // Public function that returns an array of NoteDTO 
        // (see this DTO definition below). This will be 
        // useful for client queries (Scripts), which are 
        // read-only, and we cannot just return a Resource. 
        pub fun allNotes(): [NoteDTO] {
            // We get a NoteDTO array from the note Resources 
            // in the Notepad. 
            var allNotes: [NoteDTO] = []
            for key in self.notes.keys {
                allNotes.append(self.note(noteID: key))
            }

            return allNotes
        }
    }

    // Note Resource definition. This is a nested Resource inside the Notepad Resource.
    pub resource Note {
        pub(set) var title: String
        pub(set) var body: String

        init(title: String, body: String) {
            self.title = title 
            self.body = body
        }
    }

    // Helper DTO (Data Transfer Object) that will be used 
    // when returning notes data to the clients from queries 
    // (Scripts). We cannot just return Resources there.
    pub struct NoteDTO {
        pub let id: UInt64
        pub let title: String
        pub let body: String

        init(id: UInt64, title: String, body: String) {
            self.id = id
            self.title = title 
            self.body = body
        }
    }

    init() {
        // Right after the contract is deployed, we initialize 
        // this var to keep the count of the Notepads that 
        // were created using this contract: 
        self.numberOfNotepadsCreated = 0
    }

    // Public method to create and return a Notepad Resource.
    pub fun createNotepad(): @Notepad {
        self.numberOfNotepadsCreated = self.numberOfNotepadsCreated + 1
        return <- create Notepad()
    }

    // Public method to destroy a Notepad resource from the 
    // user's Account.
    pub fun deleteNotepad(notepad: @Notepad) {
        self.numberOfNotepadsCreated = self.numberOfNotepadsCreated > 0 ? self.numberOfNotepadsCreated - 1 : 0
        destroy notepad
    }
}

Next, let's deploy the contract by tapping on the Deploy or Redeploy button at the top-right corner of the Playground:

Once the contract is deployed to the 0x01 account, you will see it in the sidebar:

Now it's time to test our contract using Transactions and Scripts. Let's start by adding a new transaction to the sidebar. This transaction will create the needed notepad and test the API by adding, editing, and deleting notes:

// Import the Smart Contract from the user account it was deployed to:
import NotepadManagerV1 from 0x01

transaction {

 // In the prepare function, we have access to the private 
 // space of the user's account who's signing the transaction:
    prepare(acct: AuthAccount) {
        // We try to borrow the notepad from the storage place 
        // where it is stored. Borrowing here means that we 
        // are creating a reference to the notepad without 
        // actually moving it to a var from its location:
        var notepad = acct.borrow<&NotepadManagerV1.Notepad>(from: /storage/Notepad)

        // If the result of that borrow is nil, that means we 
        // didn't store a notepad yet there, so we do it:
        if notepad == nil {
            // Create a notepad using our contract and move 
            // the returned resource to the /storage/Notepad 
            // path:
            acct.save(<- NotepadManagerV1.createNotepad(), to: /storage/Notepad)

            // Make the notepad public by creating a public 
            // link stored in /public/PublicNotepad. We are 
            // making it public because we need to expose it 
            // if we want to query and return that info from a 
            // Script later. These 'link' and 'borrow' 
            // functions are related to Capability-based 
            // Access Control, which is a pretty powerful 
            // feature that lets you expose a public API over 
            // the fields and functions of your stored assets 
            // to other users. If you want to dig down deeper 
            // on the topic, please refer to
            // https://docs.onflow.org/cadence/language/capability-based-access-control/:
            acct.link<&NotepadManagerV1.Notepad>(/public/PublicNotepad, target: /storage/Notepad)
            log("Notepad created!")
        }

        // We borrow the notepad (in this case we are sure we 
        // have created it because of the previous code):
        let theNotepad = acct.borrow<&NotepadManagerV1.Notepad>(from: /storage/Notepad)

        // Test note creation:
        theNotepad?.addNote(title: "First note title", body: "This is the body of the first note.")
        log("The first note was created and added to the notepad!")

        // Test another note creation:
        theNotepad?.addNote(title: "Second note title", body: "This is the body of the second note.")
        log("The second note was created and added to the notepad!")

        // Test note edition by editing the second note:
        theNotepad?.editNote(noteID: 2, newTitle: "Edited second note title", newBody: "EDITED: This is the body of the second note.")
        log("The second note was edited!")

        // Test note deletion by deleting the first note:
        theNotepad?.deleteNote(noteID: 1)
        log("The first note was deleted!")
    }

    execute {
        // We don't need to execute anything else for our 
        // testing. You can even delete this method. 
        log("Transaction executed!")
    }
}

To execute this transaction, tap on the Send button in the top-right corner of the Playground, but make sure you add the 0x02 account as the signer, so you test the transaction from a different account from where the contract is located (0x01):

After sending it, it will be executed, and you will see the logs in the Transaction Results console:

If you enable the Resources Explorer by tapping on the database icon next to the 0x02 account in the sidebar, you will see an empty storage panel above the Transaction Result console:

The notepad containing the notes should appear there. This is a recent bug in the Playground they haven't fixed yet. It was working in previous versions. But don't worry, we can check the content with a script.

So let's create a new script in the sidebar and paste this code:

import NotepadManagerV1 from 0x01

// A script is a special type of Cadence transaction
// that does not have access to any account's storage
// and cannot modify state. Any state changes that it would
// do are reverted after execution.
//
// Scripts must have the following signature: pub fun main()
// We can specify a return type, in this case we are returning 
// an array of NoteDTO:
pub fun main(): [NotepadManagerV1.NoteDTO]? {

    // Cadence code can get an account's public account object
    // by using the getAccount() built-in function.
    let notepadAccount = getAccount(0x02)

    // Get the public capability from the public path of the 
    // owner's account.
    // Remember that we made this public in the test 
    // transaction. 
    let notepadCapability = notepadAccount.getCapability<&NotepadManagerV1.Notepad>(/public/PublicNotepad)

    // Borrow a reference for the capability
    let notepadReference = notepadCapability.borrow()

    // If the notepad doesn't exist yet publicly, we return nil. 
    // Otherwise, we return all the notes in the form of an array of NoteDTO
    return notepadReference == nil ? nil : notepadReference?.allNotes()
}

Execute the script by tapping on the Execute button on the top-right side of the Playground, and you will see the result printed in the console. The notepad containing the edited note:

20:57:37 
Query notes 
Result
{"type":"Optional","value":{"type":"Array","value":[{"type":"Struct","value":{"id":"A.0000000000000001.NotepadManager.NoteDTO","fields":[{"name":"id","value":{"type":"UInt64","value":"2"}},{"name":"title","value":{"type":"String","value":"Edited second note title"}},{"name":"body","value":{"type":"String","value":"EDITED: This is the body of the second note."}}]}}]}}

Finally, if you want to easily reset all data stored in the Playground accounts, you can do it by redeploying the contract.

Link to the full Playground

That is enough for now! I hope you have enjoyed playing with Flow. In the next article, we'll learn to deploy the contract to the actual testnet, create a native iOS dApp that leverages all of this, and communicates directly with the Flow's network. Link to the second part:

How to build a native iOS dApp that uses the Flow Blockchain as the backend

Thanks for reading!

Building your next application using blockchain: Why and how

Adrián Lorenzo — Tue, 14 Jun 2022 09:23:33 +0000

Why build an application using blockchain?

The first question someone could have when considering building applications using blockchain is why doing it. It is a very important question that we believe should be asked no matter the technology or the solution that you are trying to create, since usually our objective is to solve a problem, not to use a specific technology.

Nowadays, the Web3 ecosystem is very focused on very specific use cases, but that doesn’t mean that those specific use cases are the only ones that blockchain can provide value to. I believe that the value that blockchain can provide is beyond digital assets and decentralized finance. I believe that trust is the most important value that blockchain can provide.

Trust as the value

Any individual or entity that wants to make use of a solution needs a certain amount of trust to make sure that it is going to work as expected. Sometimes this is easy to provide, but sometimes there are many environmental factors that can reduce the amount of trust that someone can have in your application. If your user has too much to lose by using your application, then they may not want to use it.

If you think that your solution is missing mechanisms to provide users with trust, it is possible that blockchain can provide those mechanisms, since it enhances trust by offering security, continuity, and transparency.

Be safe, then I can trust

Security is one of the main concerns that people currently have when it comes to building software. Without security, you can’t obtain trust, and therefore, many of the solutions that you could think of don’t make sense for the user, since they don’t really solve the problem. When we create an application that provides value to the user, you don’t want the rules of the games to be violated by an external party (hacker) or an internal party (team insiders, other users, etc.), because it means that the value that you offer to the user could be lost forever.

And when we talk about value, we need to talk about data. The internet and all the applications that we use nowadays are driven by data, and it even serves as the main way of generating revenue for some companies. That is why protecting the data that drives your application is essential.

Current blockchains have been designed with security as their main objective. If you don't trust any of the individual actors within a platform, it doesn't mean that you can't trust the system as a whole. The system has been designed to give more power to those actors who have a lot more to lose by being dishonest.

Expect continuity, then I can trust

If the data that is hosted in your application is not guaranteed to be preserved through time, many parties involved in the application have to accept the risk of potentially losing valuable data, which could decrease user trust.

Blockchain transaction records immutability allows users to keep track of every data entry, forever. You can expect your data to always be stored in the system, or for it to be stored as long as predicted.

Be transparent, then I can trust

In terms of the business logic rules of the game, when many parties are involved, it is essential for cooperative solutions to keep data transparent, so people can trust each other and therefore, be confident when making use of the cooperation tools provided.

If the application you are using is also being used by many different parties that you don’t trust, you can verify if dishonest participants are using the application maliciously. This would therefore affect the value that the application is providing to you.

Blockchains allow users to have a full history of the actions that are committed to the system, without the possibility of modifying previous records.

Smart contracts

Smart contracts are a solution provided by general-purpose blockchains (also known as layer-1 solutions) to program limited applications that run and store their data inside the network nodes. They are usually executed in a virtual machine. Currently, the Ethereum Virtual Machine is the most popular one.

You can use programming languages like Solidity or Vyper to program smart contracts that run in the Ethereum Virtual Machine.

// SPDX-License-Identifier: MIT
pragma solidity ^0.8.13;

// Example contract in Solidity
contract Storage {
    uint256 value;

    function store(uint256 _value) public {
        value = _value;
    }

    function get() public view returns (uint256) {
        return value;
    }
}

This solution is the response of these general-purpose blockchains to those use cases that would benefit from the network effect of the blockchain. The advantages of this mechanism are pretty clear:

The application takes full advantage of the network effect.
You can interact with other smart contracts in the ecosystem.
The application is hosted in the same ecosystem as any other app, reducing the entry barriers for users to use it.

Smart contracts are great, but not always enough

However, smart contracts have some critical disadvantages that can cause huge drawbacks for many solutions:

It is not scalable

Being in the same environment as everyone is a great advantage in a small number of use cases. For example, solutions that are interconnected with many other “public-good” applications and have complex interactions (more than just querying) between them. Also, the security advantages are important for certain use cases.

However, most applications do not take advantage of this, since the dynamics of an application are usually limited to a small set of interactions, or don’t need such an amount of security granularity.

The thing is that usually, the most used general-purpose blockchains, those that have the best network effect, are VERY secure. So secure, and so decentralized that they are often very slow and not scalable. This problem is generally called The Blockchain Trilemma.

Many different smart contracts are being executed, and therefore, requesting the same amount of security and decentralization as any other, even if their use case is not taking advantage of that level of security. The consequence of everyone taking advantage of the maximum network effect is a saturated network that is not able to scale when the amount of users running transactions is unsustainable.

Your application could be undestroyable and super-interconnected, but if one million requests per minute make your application unusable for those users, maybe you are not as interested in that amount of security and interconnection as you thought.

It is not private

Taking full advantage of a public network has the disadvantage of also submitting your data to the public domain.

To have a decentralized application doesn’t necessarily mean that you want to let everyone know about your data and let them participate in your network. In many organizations or cooperatives, individuals could have common objectives but also common secrets that they don’t wish to share with anyone other than the participants of the organization. Ideally, everyone should be able to be a participant in the system they trust.

It is limited

Smart contracts offer a very limited set of instructions that can be run. It is much more limited than programming your own traditional application, for example, in the Java Virtual Machine (JVM).

This limitation means fewer attack vectors, but also means less configuration and more work in many cases.

Application-specific blockchains

Instead of making use of a general-purpose blockchain to build your application on top of it using smart contracts, a great alternative could be to build your own blockchain, as your application.

An application-specific blockchain is just a blockchain that understands and provides operations and queries for your business logic. It uses the necessary resources that your application needs to run, and is configured to match the requirements that your application has.

Since your blockchain is being used for just one purpose, it means that your resources are being used for just that purpose, and therefore it is maximizing performance. Also, the system can be customized as you want to enable the best security, decentralization, and scalability configuration for your use case.

In addition, since you are building everything “from scratch” and your business logic is not running in a limited set of instructions, you are free to make use of any of your programming language tools.

Independent networks: to be isolated or not to be?

Having a custom network that runs independently from other networks has many advantages, but it also has disadvantages that we can’t ignore.

If your application is hosted as an independent network blockchain, it means that it is not going to take advantage of the network effects that a major network is going to provide, like security and decentralization.

Also, if your system wants to communicate with other networks, those networks should understand the same language as your own, and therefore, have a standard communication interface to make use of.

Permissioned blockchains

In some cases, this isolation is something that your requirements may take advantage of. If you don’t want everyone to be a participant of the network, a permissioned blockchain can match the requirements of your network.

Taking advantage of network effects with interoperability solutions

Now, imagine that you could have an independent network that could make use of some of the advantages of a major network without losing its independence. To allow that, there should be a mechanism that enables interconnection between those independent networks so they can fulfill their wishes.

Blockchain interoperability solutions allow independent networks to take advantage of at least part of the network effect of the major network without compromising scalability or configuration.

There isn’t one single way of enabling blockchain interoperability, and the way the mechanism is architected offers different capabilities to the solution you want to create.

Sidechains

A sidechain is a blockchain network that is interconnected to a bigger network, usually called the “mainchain”, following a chief/worker pattern.

The sidechains and the mainchain communicate with each other with a specific protocol. The sidechains have to validate their state with the mainchain periodically to remain part of the network, by submitting proofs. This allows the sidechain to have an extra layer of security by obtaining direct validation from a bigger network and therefore, taking advantage of its network effect.

Some key characteristics of sidechains are:

The main network is never affected by the problems that sidechains could face.
The sidechain doesn’t need to be public or have the same implementation as the mainchain.
Since you don’t have to validate each sidechain’s transaction with the mainchain, the mainchain is able to scale more effectively.
The mainchain can act as a bridge between sidechains to communicate if necessary.

Inter-communication protocols

Another solution for better scalability through interoperability is the implementation of inter-communication protocols. One of the most popular inter-communication protocols is the Inter-Blockchain Communication Protocol (IBC).

This protocol allows chains with similar core configurations to interact with each other.

Sharding

Sharding is the database architecture concept of dividing a database horizontally into independent databases, called shards. Each shard has different data and it is usually located in a different server, to allow better scalability.

The sharding concept is being introduced in blockchain networks too: the idea is to create different shards (subnetworks of nodes) that store a part of the blockchain. Each shard is going to be able to communicate with other shards and participate in the network as one.

Since no one should store the whole state and receive transactions to obtain transaction finality, the network is going to be able to scale more effectively.

Bridges

A bridge is a mechanism to interconnect blockchains through an “information exchange” using an intermediary. There are two types of bridges:

Trusted bridges: based on trust assumption, and handled by a centralized entity.
Trustless bridges: based on an algorithm, usually written in a smart contract.

A bridge is made specifically for interconnection between a set of specific blockchains with specific configurations. It doesn’t act as a protocol.

Does this mean that I should discard smart contracts?

Smart contracts are VERY useful for many use cases. There are a lot of applications like Uniswap, Filecoin or Developer DAO that make an excellent use of smart contracts to implement their applications.

In this article I have analyzed some of the current advantages and disadvantages of building smart contracts and application-specific blockchains. Depending on your use case, you may find one of the solutions more suitable than the other.

Wait, should I implement application-specific blockchains from scratch?! 😦

You could actually implement any of these mechanisms and blockchain specifications from scratch, but you probably don’t have the time and the resources to even consider it.

However, we are in luck. There are a lot of useful open-source tools that allow developers to build application-specific blockchains without having to worry about low-level implementations and cryptographic details. This is done by making use of industry-standard specifications and protocols backed up by the community.

In my next article, I’m going to review some of the most relevant solutions that you can find in the market right now.

Stay tuned!

This article represents a very small piece of what we are building at The Agile Monkeys. If you like what we do, you may want to join us in our journey!

Photo by Shubham Dhage on Unsplash

Playing the "no game" in meetings.

David Godoy — Tue, 07 Jun 2022 11:35:10 +0000

Ok, I need to confess something. I have lost uncountable hours in meetings during my career. Not only because they could have been an email or not needed at all, but for things that seemed reasonable. Those hours spend on legit meetings are always justifiable, and we see them as necessary but sometimes they take more than needed.

Don't get me wrong. I'm not saying those meetings were a complete waste of time. My point is that sometimes those meetings are missing something essential to be effective, The Objective™. Being in a meeting without a clear objective is like playing a game without knowing the rules or the goal you need to reach; you are playing the “no game” game. Without it, something that could take 10-15 minutes is "done" in 40, people in the meeting lose focus, and your energy level after the session is sea bottom level.

The “No game” game

This is especially relevant with teams with very different skills and backgrounds. When you don't know which game you are playing, you just play the game you are more comfortable with. Developers play code, Managers play hours, DevOps play security... you know "their games". Even when the objective is mentioned in the event's email or at the start of the meeting, someone can start playing the "no game", and our natural tendency is to play with them. This is not related to the team experience or excellence, I have been in meetings with brilliant and capable people, and we have started to play the "no game" pretty heavily.

Let me share three real-world situations I have been involved in and how noticing we were playing the "no game" game saved us from the bottomless time pit.

Unblock the release

We migrated a mobile app backend from Firebase to Booster framework to support multi-cloud, and use some exciting features (event-sourcing, serverless, etc.). Hence, we needed to update the app code to use the new GraphQL interface. We updated the iOS application and applied some design changes along the way. Still, the Android app was a bit behind on the update, so we decided to have a meeting to discuss how to unblock the situation and release the new app as soon as possible since it was impossible to release only one version.

The objective was evident when we started the meeting, but it didn't take much to start playing the "no game" game. We discussed the need to update the Android app to be on the latest and greatest software stack to make it easy to keep the iOS pace in the future. We mentioned that the app was old, and giving it a good refactor makes total sense. After some time, we started talking about taking two more months to release it, and as expected, the product owner was a bit concerned and started asking.

Fortunately, someone from the team grabbed us back into the "right" game, remembering that the goal was to unblock the release, not update the Android app to the same point as the iOS. After that, we all agreed on a plan that could be executed in a few weeks. We ended the meeting there, and everyone could keep going with their stuff. This saved us a lot of time and frustration on the team.

The Football game

Unit testing a Proof of concept project

At that time, we were doing some product research, and we had a potential client we wanted to show some concepts to and discover if they were appealing to them. The first meetings were tremendous, and they wanted to keep exploring the solution, so we decided that we could present them with a "Proof of Concept" application to engage even more with the product.

We had a meeting to coordinate and plan that "Proof of Concept". Since our team was excellence-focused and we were all passionate engineers, someone started talking about adding tests so the future refactors and maintenance were easier. We would create it from scratch, and we could enforce them in every PR. That way, we could ensure the highest code coverage possible. If you have been in this situation, you know that early testing debates can last forever. Someone mentioned that our goal was to have something pretty and functional to show to the client quickly. After that, we all agreed to enforce the tests later on when building the real thing. It seems trivial, but by clarifying our objective, the game got so clear that we quickly create a plan and finished the meeting.

The Squid game

Brainstorming too specific

We were throwing crazy ideas to the table in a brainstorming session, and suddenly we started going way too far with the reality checks. We started designing a specific solution for that idea and arguing about different approaches. It is always lovely to debate and get passionate about implementation details, but sometimes it is outstanding that someone just raises their hand and remembers the game we are playing. It saves time and prevents going too deep into the design of something that will change or be dismissed by another idea.

The Tennis game

With no effort, you can probably tell similar situations you have been involved in, and they took ages to finish with probably no clear outcome. The worst part is not the time you waste arguing in the meetings but the frustrating feeling of talking with a different game in mind. Whether is a manager, team lead, or teammate, clarifying the objective is always helpful.

From my experience, the biggest takeaways from those situations are the following:

Always communicate the objective of the meeting
Having an objective in mind removes noise and brings you clarity for the decisions to make
Sometimes you need to remember the objective of the meeting. A.K.A.: The "game" you are playing.
Have the main objective clear. If other games need to be played, it is vital to do it at the right time (or meeting).
Even when you agree with the given statements, if they are playing a different game, the best thing to do is remember the objective and leave it aside for a moment.
Great leaders always try to keep the objective in mind and fresh to help the team be on the same page and focus on the current game.

Knowing the game you are playing is key to being practical. It is a positive spiral that allows you to waste less time, have less anxiety/frustration, solve problems better and faster, and make things 10x better.

Bugs will happen. Are you prepared?

Jorge Rodríguez — Thu, 02 Jun 2022 11:02:02 +0000

It's very common for developers (and managers!) to focus on new features, performance, or bug fixing when coding. But when a project is complex enough, there's a hard truth that we all must take into account: bugs will happen. Period. I have never seen a new project go live without issues. In fact, I don't think I've ever seen a project stay bug-free in production for more than a month straight. Software engineering is imperfect, and our code must be prepared to deal with it. This post is about what you can do to make life easier for you and your team.

A couple of years ago, a bunch of us at The Agile Monkeys started this huge project for a bank. We rebuilt a critical part of their system, with an event-driven architecture that would be processing millions of monetary (and non-monetary) transactions daily. The team did a fantastic job and it's still in production and working well to this day. However, it was not a bed of roses. We faced some important challenges that I believe made us all much better developers today.

As usual with any big project, there was a lot of technical learning involved. From do's and don'ts in event-driven architectures, to communication both inside and across the teams. There were loads of lessons learned. But there was one huge takeaway that I took from the project. We dedicated way too much time to code optimization, bug fixing, and new features. And we didn't work enough on defensive programming.

What's defensive programming?

Quoting Wikipedia directly:

Defensive programming is a form of defensive design intended to ensure the continuing function of a piece of software under unforeseen circumstances. Defensive programming practices are often used where high availability, safety, or security is needed.

The emphasis here is on unforeseen circumstances. Bad Things™ will happen, no matter what you do. So you have to prepare for them.

At our first launch, we thought that we had a very robust application. It was performant and well tested. But, again, Bad Things™ can and will happen. In our case, it was an unknown rate limitation on our vendor production API that temporarily shut our application account down. Then, even when our services had a retry mechanism in place, they all ended up retrying simultaneously. As a result, a large majority of our transactions failed. Recovering from those errors required a lot of work from all of us. We were clearly not prepared enough for an issue at such a scale. We found ourselves doing a lot of manual, tedious work to solve the transactions in pending status. It resulted in many hours of hard work, and it naturally grew into some automatic tooling for reconciliation. Tooling that we should have implemented right away, instead of dedicating that much time to optimize performance, fixing bugs or even adding new features.

Reconciliation tooling

In regards to the tooling that you should have in place, it will greatly depend on the needs of your system. But for an event-driven architecture like the one that we were working with, we found it incredibly useful to have an Admin API (be careful with the security on this one!), with some automatic checks and reconciliation logic. The methods in this API would be able to replay events, look for inconsistent values in DB, or for the most extreme scenarios, even update some DB fields directly.

That, combined with an event store where every event received is kept (especially if not completed successfully), will allow your team to work wonders. It's important that the developers are able to safely and consistently replay the events. One of the issues that we faced in our project was that, even though our log aggregator worked very well to grab event data, it was not consistent enough for our scale. A small percentage (below 1%) of events were lost, but it equated to thousands of events that need to be manually searched for. Having a more complete and reliable event store for our needs was a huge relief for our reconciliation process.

There's another very good benefit that you'll get from all of this tooling: you can automatize those checks and processes to run periodically once you've verified their robustness. For instance, you could run a reconciliation task daily on all of those events that failed or stalled for more than an hour. Once the tooling is in place, there are many advantages that you can enjoy!

So, for every reader out there, I'll be proud if you can just carry home the following takeaway: bugs will happen. Are you prepared to deal with them in production? If not, start dedicating some time to it, even before bug fixing or optimizing your code. Invest in a calm mind. Future You will thank you when enjoying the spare time with your friends instead of dealing with on-call issues!

Logging Ethereum events with NodeJS

Roberto Diaz — Wed, 25 May 2022 22:50:59 +0000

Believe it or not, Web3 and distributed applications are here to stay! These technologies are contributing to creating a more decentralised Internet where the data is not owned by single institutions and applications are not necessarily running on servers managed by big companies. We are talking about a utopic future where data and applications are not attached to any particular place, they just will exist in the networks.

Until that utopic point, traditional cloud applications are going to coexist with distributed applications and in some use cases, they will need to interact between them. Cloud applications could need to read transactions around a specific address, check that a smart contract fires a specific event or simply send a transaction or data to a specific address, among other interactions.
We are going to cover the necessary pieces to achieve that a nodeJS application will be able to interact with the Ethereum network with some simple examples.

The Intermediary

The first actor in this integration between the traditional nodeJS application and the Ethereum network is the mechanism that will allow us to listen to the blockchain and send data to the blockchain.

To have this communication mechanism we have two options, create an Ethereum node or use a third-party provider. The main difference between the two options is confidence.

If we manage our node, we know that information is not managed by a third party and we can trust the data that we are reading. On the contrary, the drawback of managing your node is that normally it is more complex than using a third party to connect to the blockchain network.

To mount an Ethereum node, the recommended option is to use Geth. Geth is one of the three original implementations of the Ethereum protocol implemented in Go and it is one of the most used implementations to create an Ethereum node. You need a machine with A LOT space in your hard drive, and download, configure and launch Geth.

Another alternative to mounting your node is to use a cloud provider such as AWS or Azure. In cloud providers, you can mount Geth on a normal cloud instance or use specific services such as Amazon Managed Blockchain or the Ethereum node from Azure Marketplace.

If you don’t want to deal with the configuration of an Ethereum node you can use a third-party provider that can provide you access to the network through them. Some of the most well-known providers are Infura, Alchemy and QuickNode. In our example, we are going to use Infura.

Finally, not covered here, another option to interact with blockchain could be to use the Etherscan API to read information about accounts, transactions, blocks, etc… using a REST API.

The Javascript interface

From the previous step, using a self-owned node or a node provided by a third party, we are going to obtain an URL to a JSON gRPC API.

To interact with the Ethereum node we are going to use a JSON gRPC API but fortunately, we won’t need to interact directly with it. Several libraries work as a wrapper around the node API making our life easier. Web3js and Ethers are two of the most commonly used. In our examples, we are going to use Web3js.

At this point, we have all the pieces in place to start to connect nodeJS with Ethereum and start to receive events and monitoring addresses. The following diagram summarises what we have said.

Reading information

Once we have everything ready in our NodeJS application, we can start to receive information from the Ethereum network. Web3js provides us with some convenient methods to receive information depending on what we want to monitor. If we want to monitor what’s going on around a smart contract we are going to use the subscribe method. This method allows us to receive every event emitted by a smart contract.
An important consideration here is that with emitted events we are talking about Solidity logging events. If your contract doesn’t emit any event you won’t see any data using the subscribe method. Here is an extract of code using the subscribe method:

const options = {address: '0xcontract_address'};
var subscription = web3.eth.subscribe('logs', options, function(error, result){
 if (!error) console.log('got result');
 else console.log(error);
}).on("data", function(log){
 const decodedLogs = abiDecoder.decodeLogs([log]);
 console.log(decodedLogs)
}).on("changed", function(log){
 console.log('changed');
});

In this fragment of code, we are simply subscribing our application to Ethereum logs of a specific contract address provided in the options object. Additionally, we are decoding the event using the library abiDecoder. To use this library, we are providing in the configuration the Contract.json file including the ABI (Application Binary Interface) of the contract.

Using Web3js we can subscribe our application to specific events using the configuration of the topic. In this case, we are subscribed to all the events emitted by the contract. Here is an example of an event received from our test contract after emitting a transfer:

Inside the event array we get more information about the emitted event:

Get more information from the network

At this point, we can monitor a smart contract but we can try to go further and try to monitor what’s going on around a specific address. Thanks to Web3js we can also subscribe to every block added to the blockchain and we can explore the content of that block.

let blockSubscription = web3.eth.subscribe('newBlockHeaders')

blockSubscription.subscribe((error, result) => {
 if (error) {
   console.log("Error subscribing to event", error)
   process.exit()
 }
}).on('data', async blockHeader => {
 if (!blockHeader || !blockHeader.number)
   return
 const block = await web3.eth.getBlock(blockHeader.number)
 console.log(block)

})

In this case, we are just subscribing our application to the newBlockHeader and printing the result:

In the block header information, we have a lot of information about the latest mined block but there is a very important section for us, the transactions.

That array includes all the hashes of the transactions included in that block and again thanks to Web3js we have a method to obtain the information of a specific transaction. As you can see, if we want to monitor just a subset of addresses now is where the intensive processing stuff starts XD.

const block = await web3.eth.getBlock(blockHeader.number)
 block.transactions.foreach( transactionHash => {
   transaction = await web3.eth.getTransaction(transactionHash)
   console.log(transaction)
})

As a result of this we can get all the information related to the transaction:

We can get the from and to addresses, any additional information included inside the transaction in the input field or the amount of the transaction in wei inside the value field.

DISCLAIMER: As you might be guessing, we are performing a lot of requests against the network, for each received block, we are requesting the block information and then we are asking for the information about all transactions included in that block. Because of this, we could reach the limit of usage of a third party node such as Infura. For this specific use case, it could be recommended to have a self-managed node.

For more information about configurations and to check how it works, all the code shown can be found in this repository. The repository includes a simple smart contract developed with Hardhat and all the necessary code to monitor the contract and some Ethereum addresses using NodeJS.

Cover Photo by Shubham Dhage on Unsplash

DEV Community: The Agile Monkeys

Navigating the Double-Edged Sword of AI as a Software Developer

Introduction

Great Power...

Great Responsibilities

How I Use AI in My Daily Job

What is Coming and How to Maximize the Advantages

Conclusion

Baby's First Contribution - A GitHub Flow for non-devs

Get to meet Git - Time Machine for code

GitHub - A Google Drive for Git logs

Making the change

Creating a branch

Doing the actual changes

Creating a Pull Request

Congratulations, you're officially a nerd programmer now!

Railway Event Processor: Safe integrations in Event Sourcing

Railway Event Processor

When in doubt, treat your system as distributed

Modelling systems as a workflow of tasks

The Task

To-Do Item

State

The transition

Technical categories of tasks

External action required

Idempotency enabled

Optimistic lock enabled

Pessimistic lock enabled

Black boxed

Common language definitions

To Do

Trivial

Deterministic

Opaque

Business concerns of a Black Boxed Task

Visibility

Necessity

Concern

Risk

Compromise Decision of Black Boxed Tasks

At least once

Up to n times

At most once

Important: Human intervention required

The conclusion

Algorithms for each kind of Task

Every task requires:

Task identifier

Expiration functions

To-Do Item

State

Retriability

Fault tolerance

Idempotency enabled

Optimistic lock enabled

Pessimistic lock enabled

Black boxed

The Workflow

The core tenets of the framework

Rejection is not failure

Undo "A" is just do "B" when "A" has been done

All parties are third parties

Yes, even people are third party services

Insist, desist or react

Optimism through deterministic pessimism

The framework, by example

Legend

Glossary

Code safety:

Completed task:

Concurrency handling:

Failed task:

Fault tolerance:

Optimistic lock enabled action:

Pessimistic lock enabled action:

Coaches don’t play

You are not a player anymore.

The delivery trap

Make yourself redundant

Up to `n` times

`FlowNotesApp.swift`

`LoginView.swift`

`LoginVM.swift`

`MyNotesView.swift`

`MyNotesVM`

`NoteEditorView.swift`

`Note.swift`