DEV Community: Martin Beierling-Mutz

How to Consistently Retrieve Valid JSON from Claude 3.5 in Go

Martin Beierling-Mutz — Thu, 17 Oct 2024 12:19:52 +0000

When working with LLMs as a developer, you often want to receive data in a structured format. While this didn't always work well in the early GPT-3 era, the current models that support function calling are much better at it.

In this post, I want to share a simple function called CallClaudeForceTool, which you can use to call Claude to receive structured data/JSON output. In my case, it is written in Golang and returns any type of struct I need.

First, you need to define the structure and provide it to the LLM in the format of a JSON schema. In my example, I’m defining a struct type that serves as the input to a function that queries for some data.

Here is the struct that is the input to the query function:

type QueryArgs struct {
    SearchQuery string `json:"searchQuery,omitempty"`
    TypeFilter  string `json:"typeFilter,omitempty"`
}

The following is the JSON Schema definition. Adding good names & descriptions is actually important here, as the LLM will use this information when generating the JSON data:

var QueryFnSchema = JSONSchema{
    Type: ai.DataTypeObject,
    Properties: map[string]ai.JSONSchema{
        "searchQuery": {
            Type:        ai.DataTypeString,
            Description: "The search text query to use. Any data that contains this text will be searched. If not provided, then no text comparison will be performed.",
        },
        "typeFilter": {
            Type:        ai.DataTypeString,
            Description: `The type to filter on. If not provided, data of any type will be searched. Can only be one of: "location", "event", "person", "organization".`,
            Enum:        []string{"location", "event", "person", "organization"},
        },
    },
}

type JSONSchema struct {
    Type        DataType              `json:"type,omitempty"`
    Description string                `json:"description,omitempty"`
    Enum        []string              `json:"enum,omitempty"`
    Properties  map[string]JSONSchema `json:"properties,omitempty"`
    Required    []string              `json:"required,omitempty"`
    Items       *JSONSchema           `json:"items,omitempty"`
}

func (d JSONSchema) MarshalJSON() ([]byte, error) {
    if d.Properties == nil {
        d.Properties = make(map[string]JSONSchema)
    }
    type Alias JSONSchema
    return json.Marshal(struct {
        Alias
    }{
        Alias: (Alias)(d),
    })
}

type DataType string

const (
    DataTypeObject  DataType = "object"
    DataTypeNumber  DataType = "number"
    DataTypeInteger DataType = "integer"
    DataTypeBoolean DataType = "boolean"
    DataTypeString  DataType = "string"
    DataTypeArray   DataType = "array"
    DataTypeNull    DataType = "null"
)

I’m using a simple JSON Schema struct. You could use a dependency for that, but this struct is usually sufficient for LLM function calling. It works great with Anthropic’s Claude & OpenAI’s ChatGPT.

You would define your own schema instead of QueryFnSchema, based on your equivalent to QueryArgs or whatever data you need and want the LLM to respond with.

Now it’s time to use the CallClaudeForceTool function (which will be explained at the end of the post):

func main() {
// ...
    queryArgs, claudeResponse := CallClaudeForceTool[QueryArgs]([]ClaudeLLMMessage{
            {Role: "user", Content: getQueryPrompt(inputMessage)},
        }, ToolDefinition{
            Name:        "search",
            Description: "Search for data in the database",
            InputSchema: queryFnSchema,
        }, ClaudeConfig{
            MaxTokens:   4000,
            Temperature: 0.8,
            System:      SYSTEM_PROMPT,
        })

    result := db.Query(queryArgs)
// ...
}

You’d replace the getQueryPrompt(inputMessage) , SYSTEM_PROMPT and any config with your own needs.

The queryArgs variable now contains the LLM-generated data in the format you defined, which is the QueryArgs struct in my case. At this point, I use it as a parameter for the actual function call to query my database. However, you can also generate any structured data from some input or prompt and then use it in your code.

Finally, here is the CallClaudeForceTool function and any auxiliary types. Its goal is to call Claude, force it to use the single tool provided, and end without responding to Claude with any subsequent tool/function output or receiving any final response text message. We just want Claude to respond once with the structured data.

func CallClaudeForceTool[T any](messages []ClaudeLLMMessage, tool ToolDefinition, config ClaudeConfig) (T, *ClaudeCreateMessageResponse, error) {
    payload := ClaudeCreateMessagePayload{
        Model:    "claude-3-5-sonnet-20240620",
        Messages: messages,
        Tools:    []ToolDefinition{tool},
        ToolChoice: ClaudeToolChoice{
            Type: ToolChoiceTypeTool,
            Name: tool.Name,
        },
        MaxTokens:   config.MaxTokens,
        Temperature: config.Temperature,
        System:      config.System,
    }

    payloadBytes, err := json.Marshal(payload)
    if err != nil {
        return *new(T), nil, fmt.Errorf("impossible to marshal payload: %w", err)
    }

    req, err := http.NewRequest(
        http.MethodPost,
        "https://api.anthropic.com/v1/messages",
        bytes.NewReader(payloadBytes),
    )
    if err != nil {
        return *new(T), nil, fmt.Errorf("impossible to create request: %w", err)
    }
    req.Header.Set("x-api-key", os.Getenv("ANTHROPIC_API_KEY"))
    req.Header.Set("content-type", "application/json")
    req.Header.Set("anthropic-version", "2023-06-01")

    // send the request
    httpClient := http.Client{Timeout: 60 * time.Second}
    res, err := httpClient.Do(req)
    if err != nil {
        return *new(T), nil, fmt.Errorf("impossible to send request: %w", err)
    }
    defer res.Body.Close()

    resBody, err := io.ReadAll(res.Body)
    if err != nil {
        return *new(T), nil, fmt.Errorf("impossible to read all body of response: %w", err)
    }

    // unmarshal the response
    var response ClaudeCreateMessageResponse
    err = json.Unmarshal(resBody, &response)
    if err != nil {
        return *new(T), nil, fmt.Errorf("impossible to unmarshal response: %w", err)
    }

    // Get the tool response
    var toolInput T
    for _, content := range response.Content {
        if content.Type != ClaudeResponseContentTypeToolUse || content.Name != tool.Name {
            continue
        }
        inputBytes, err := json.Marshal(content.Input)
        if err != nil {
            return *new(T), &response, fmt.Errorf("impossible to marshal tool response: %w", err)
        }
        err = json.Unmarshal(inputBytes, &toolInput)
        if err != nil {
            return *new(T), &response, fmt.Errorf("impossible to unmarshal tool response: %w", err)
        }
        return toolInput, &response, nil
    }

    return toolInput, &response, fmt.Errorf("impossible to find tool response")
}

// Auxiliary types

type ClaudeLLMMessage struct {
    Role    string `json:"role"`
    Content string `json:"content"`
}

type ClaudeConfig struct {
    MaxTokens   int
    Temperature float64
    System      string
}

type ClaudeCreateMessagePayload struct {
    Model       string             `json:"model"`
    MaxTokens   int                `json:"max_tokens"`
    Messages    []ClaudeLLMMessage `json:"messages"`
    Stream      bool               `json:"stream,omitempty"`
    System      string             `json:"system,omitempty"`
    Temperature float64            `json:"temperature,omitempty"`
    ToolChoice  ClaudeToolChoice   `json:"tool_choice,omitempty"`
    Tools       []ToolDefinition   `json:"tools,omitempty"`
}

type ClaudeCreateMessageResponse struct {
    Content []struct {
        Type  ClaudeResponseContentType `json:"type"`
        Text  string                    `json:"text,omitempty"`
        ID    string                    `json:"id,omitempty"`
        Name  string                    `json:"name,omitempty"`
        Input map[string]interface{}    `json:"input,omitempty"`
    } `json:"content"`
    Id           string `json:"id"`
    Model        string `json:"model"`
    Role         string `json:"role"`
    StopReason   string `json:"stop_reason"`
    StopSequence string `json:"stop_sequence"`
    Type         string `json:"type"`
    Usage        struct {
        InputTokens  int `json:"input_tokens"`
        OutputTokens int `json:"output_tokens"`
    } `json:"usage"`
}

type ClaudeToolChoice struct {
    Type ToolChoiceType `json:"type"`
    Name string         `json:"name,omitempty"`
}

type ToolChoiceType string

const (
    ToolChoiceTypeAuto ToolChoiceType = "auto"
    ToolChoiceTypeCode ToolChoiceType = "any"
    ToolChoiceTypeTool ToolChoiceType = "tool"
)

type ToolDefinition struct {
    Name        string     `json:"name"`
    Description string     `json:"description"`
    InputSchema JSONSchema `json:"input_schema"`
}

Some examples for what this can be used:

Calling a function, as shown in my example, which is probably the most obvious use case of LLM “function calling”.
- The full flow of the function calling feature would include actually responding to Claude with the tool/function’s output and generally having a back & forth until Claude responds with its final message.
Generating structured data for API requests: You can use the LLM to generate JSON payloads for API requests based on user input, ensuring that the data conforms to the required schema.
Data validation and transformation: The LLM can be used to validate and transform input data into a structured format, which can then be used for further processing or storage in a database.
Generally whenever you have some text & want to extract or transform it into a specific structure, without wanting to train your own custom classifier.

That’s it! Please let me know if this was useful, or maybe share how you are using LLMs in your applications.

Bis demnächst!

~ Martin

Notes:

This was originally posted on my blog at https://blog.embiem.me/unleash-your-home-hardware-processing-long-running-tasks-at-home
Any code is provided as-is, under the MIT license.
The cover image is AI generated.

Why I fell in love with tailwindcss

Martin Beierling-Mutz — Fri, 15 May 2020 20:03:42 +0000

(Photo by Cameron Venti on Unsplash)

About six months ago I first heard of tailwindcss. I looked at their website, but I was opposed when I saw all those classnames in the code. I didn't touch it back then.

A few months passed and I started seeing more and more tailwindcss mentions im my feeds. At some point I was intrigued enough to finally try it out.

The Struggle

I never used a CSS utility library before. My main workflow was scoped CSS using styled-components.

My first steps using tailwind were a struggle. I had to constantly lookup classnames in the documentation. Always asking myself whether this is really that much better than just writing the CSS myself.

The documentation is quite good though. The struggle of always looking up the classnames was eased that way. If the documentation was just a bit worse... I think I'd have switched back to styled-components.

The Realisation

Then, I started looking up classnames less and less. Especially after I found that there's a neat VSCode extension: Tailwind IntelliSense. I realized that I actually learned the basics of tailwind quite fast. It was just that comfortable feeling of using what you already know, which tried to nudge me back to writing my own CSS.

I soon split bigger components into smaller ones, like I usually would in my React applications. That's when I realized that tailwind is often less code than writing your own CSS. It's also scoped by nature... it's just classes.

Finally, I wanted to use my own color scheme and font. This is easily possible using a config file. That's when I realized that I was already creating beautiful UIs, without customizing anything. No need to think about defining good scales of font sizes, margins, break points etc. Very good defaults are already chosen for you. Just try multiple variations & pick the one that looks best.

The (Hidden) Benefits

No need to worry about naming conventions. Just hack pre-defined utility classnames into your keyboard to create beautiful UIs with ease. Everybody who is familiar with tailwind can work with it without any ramp-up time.

You only bundle what you use. As long as you don't deviate from tailwind's recommendation, any unused classes won't make it into the bundle.

Write less. Almost always, the utility classnames are shorter than the style they implement. Just use the classes grid grid-cols-3 gap-4 in you HTML. Instead of defining a class that implements its styles: .my-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 1rem; }, and then using that class my-grid in your HTML.

Know what's what. Previously, when I was writing my own CSS, I had to look at the implementation of a CSS class to know what it's doing. Now, you see the tailwind utility class and immediately know what it is there for.

Build responsive fast. Just look at the animation on tailwind's front page. Seriously, go there and watch it for 30 seconds. Building responsiveness into your UIs is super fast with tailwind.

Is It For You?

If you're anything like me, you know your CSS but are no CSS artist, then it is definitely for you.

Try it out! If you don't know on what, check out Frontendmentor's challenges. Then please let me know your thoughts.

Last but not least, it's not the only thing you'll ever use. You'll still write your own styles sometimes, especially if you need to pixel-perfectly implement a design. Using a mix of tailwindcss and styled-components/sass/less/css/whatever is actually fine! In fact, that's how you'd introduce it into an existing project: Alongside the tools you already use.

What do I think now, after my initial hesitation? Well I think it really is like tailwind... it pushes you forward, whether you realize it or not.

Use Machine Learning to filter messages in the browser

Martin Beierling-Mutz — Sun, 20 Jan 2019 19:41:50 +0000

We'll use an Artifical Neural Network to classify messages as "spam" or "no spam".

The goal is to clean up this mess:

The Problem

“Should this message be hidden or shown?”

All solutions start with a problem statement. In our case, we have a text message and want to classify it as either "spam" or "no spam".

The classic application for this problem would be email, but I chose to use the Twitch chat of big channels. These tend to be really spammy and thus hard to read, while easy to get data.

What's Twitch?

Twitch.tv is an online platform that allows to stream video and reach thousands of like-minded people on the internet. It is most commonly used to stream video of a video game and a person playing the game. The chat is the main window of interaction with the streamer, but especially in channels with a lot of viewers the chat can become very hectic and unreadable by spam of messages that are of low value.

The Solution

Filtering text messages into spam and no-spam is a binary classification problem. There are lots of ways to solve this, and I chose one: Artificial Neural Networks (ANNs).

More specifically, it will be a feedforward neural net with backpropagation. This is a very common and basic setup, where our nodes won't form cycles (hence feedforward) and all nodes will be notified about how well they performed (hence backpropagation).

If you want to jump directly to code, here's the repo. Please take all of it with a grain of salt (that's the right saying, isn't it?). I created this 2 years ago as part of my Machine Learning Nanodegree at Udacity. Lots has changed since then, like the release of TensorFlow.js.

embiem / Better-Twitch-Chat

Hobby-project using Twitch Chat, ML, Twitch-Kraken API, reactJS & firebase

A hobby project, which tries to implement a more comfortable & feature-rich chat app for Twitch PoC developed during the Capstone project of Udacity's Machine Learning Nanodegree.

This project was bootstrapped with Create React App Its guide can be found here.

3rd party libraries

brain.js - for classifying messages as "show" or "hide" using Neural Networks
firebase - to persist data and authenticate the user via Twitch
material-ui - easy & fast UI/UX
mimir - bag-of-words model
react - front-end framework <3
redux - state <3
redux-thunk - for those awesome actions that actually aren't actions
tmi.js - Twitch api & chat connection

New Clone

Basically follow the steps here.

Create the src/config.js file with firebaseConfig & twitchConfig.
Create the functions/service-account.json file Docs
Set the twitch clientId & clientSecret as GCloud env variables Docs
firebase functions:config:set twitch.client_id="yourClientID" twitch.client_secret="yourClientSecret"
for local dev, also set firebase functions:config:set twitch.client_id_dev="yourClientDevID" twitch.client_secret_dev="yourClientDevSecret"

Table

…

View on GitHub

There's also a live version of the Web App, which lets you connect to a Twitch channel to either filter it or train your own model based on liking/disliking messages.

Done having a quick look? Great, let's continue...

What's an ANN?

AI, Machine Learning, Deep Learning and similar buzzwords are hot nowadays. So let's make sure the terminology is clear: Deep Learning is a subset of Machine Learning and AI is a researching field, which may use Machine Learning. I always like to give Boston Dynamics' Atlas robot as an example: It's a product of AI and robotics research, but doesn't actually use any Machine Learning solutions.

With ANN, we describe the general concept of modeling a digital system after biological neural networks. Our brain is built of lots of interconnected neurons, which create neural networks. We know how they work: neurons receive inputs and may send outputs if the input is above a threshold.

An artifical neural network consists of nodes imitating the behavior of neurons. Like Lego bricks, we assemble them in specific ways to form something cool, like a Millennium Falcon

Jokes aside, it is very astonishing to see how carefully built neural network architectures are enabling things like this:

In the end it all comes down to nodes. Connected in a specific way and with some logic attached, which defines how input is used to create an output.

How does an ANN work?

Let's take our feedforward neural net with backpropagation as an example. It should classify incoming text as either "spam" or "no spam". We will input our text message to the first nodes in our network. These nodes will do some calculations based on the received input and the nodes' internal state. The results are then sent to the next nodes. This happens until we reach the last node in our network, which will be responsible for classifying the input text message as "spam" or "no spam". During training, we know whether a message is spam or not and will give the network a 👍 or 👎, based on how well it did. This feedback is propagated back through all nodes of the network and every node will tune its internal state a bit.

There's a cool playground tool which let's you train a neural net right in your browser and play around with the different parts: playground.tensorflow.org. Don't worry too much about what a learning rate or TanH activation is yet. Just play around with it a bit. Have fun 🤓

Furthermore, while writing this article, @Petro Liashchynskyi published this article, which explains ANNs and their concepts on a more technical level.

Creating of neural network using JavaScript in 7 minutes!

Petro Liashchynskyi ・ Jan 12 '19

#javascript #webdev #neuralnetworks #machinelearning

I'll also have a list of awesome ML resources at the end of this article.

Data

The most important dependency of any Machine Learning solution is data. The more and better data you have, the better will your model perform.

Data Collection

This is often one of the most difficult tasks. Just imagine, labeling millions of images with "apple", "bike", "human", "dog", ...

In (supervised) Machine Learning, the machine learns by example. So we need to give the machine lots of examples. The more complex the task (like classifying objects in an image), the more examples we need.

For our problem, we need lots of text messages and label them as "spam" or "no spam". One of the reasons why I chose to filter messages of a Twitch channel: Most messages are small and collecting data is relatively easy.

In the Web App, there's one view dedicated for collecting data. You join a Twitch channel and all messages are listed in real-time, much like the built-in Twitch chat. Additionally, there's a like/dislike button for each message to indicate whether a message is spam or not. All liked/disliked messages are sent to a database.

Data Preprocessing

One of the main differences between machines and humans: They are very efficient at reading & computing zeroes and ones, while we are incredibly good at understanding concepts. We see a bunch of letters, read them and understand the information they keep. Machines wouldn't even know what letters are.

That's why we humans build character encodings like UTF-8, which gives the machine a way to structure 0s and 1s to form a concept of letters. Then we can do things like putting multiple characters into an array to build sentences or save articles like this one in a database.

To teach a machine anything about our world, we need to keep these things in mind. In this project we deal with text, so how do we input this text to a ML model?

There's a popular solution called Bag of Words (BOW), which takes text as input and outputs a bunch of zeroes and ones. Perfect!

For this project, you can find the data preprocessing in one small file at node/dataPrep.js. Let's also get some code into this article.

First we need to take our dataset and put all messages into an array and all labels (show = "no spam" & hide = "spam") into another array:

const dataFlat = [];
const dataLabels = [];

for (let key in data) {
  dataFlat.push(data[key].message);
  dataLabels.push(data[key].liked ? 'show' : 'hide');
}

Then we create a dictionary of words, using mimir:

const dictData = mimir.dict(dataFlat);
const dictLabels = mimir.dict(dataLabels);

And finally, we optimize our data a bit by removing any entries that only appeared once in the dataset:

for (let key in dictData.dict) {
  if (dictData.dict[key] < 2) {
    delete dictData.dict[key];
    _.remove(dictData.words, w => w === key);
    _.remove(dataFlat, w => w === key);
  }
}

This is optional, but will reduce the complexity that our model will need to handle. The bigger the BOW dictionary gets, the more complex the input space. Also called: The Curse of Dimensionality.

Data Exploration & Visualization

As data is very important to the success of you ML solution, knowing your data is part of it. Without knowing how the data is structured and finding certain characteristics, it will be hard to even start developing a model. This is especially important for datasets that you didn't create.

For our project, we know the data very well as we actually created the dataset on our own by liking/disliking messages. So one interesting visualization is the word occurrence. Here's a graph showing which words occurred the most in messages labeled as "show"/"no spam":

Pretty common words, right? What's interesting is also how often these appear in messages labeled as "spam". This is interesting, because they appear very often in "no spam" messages but almost never in "spam" messages. Therefore, they have a high impact on the ML model performance when classifying messages that include these words.

This info could also come in handy when trying to reduce dimensionality. Certain approaches try to find high entropy features and combine these. But that's a story for another article.

Even if you don't know python, there's a very good notebook on Kaggle about data exploration: Comprehensive Data Exploration with Python.

For JavaScript devs, you might want to have a look at Observable. Just have a look at this data exploration JS notebook of the Titanic dataset. It's pretty cool!

Model

Here comes the easy part!

Once you know your data and what you want to achieve, it's time to create the model.

Build your model

I wasn't joking. In my opinion, data exploration actually takes more of your precious brain juice. Building and training your model "just" takes time and iteration.

You could look at research papers or a list of proven network architectures, but what fun would that be?

So let's build our model:

const net = new brain.NeuralNetwork({
  hiddenLayers: layers,
  learningRate: lr
});

I said it's easy!

This doesn't mean that it can't get complicated. It just doesn't have to. Especially with tools like brain.js, tf.js, keras etc., achieving a well performing model is possible with only a handful lines of code. This, of course, always depends on the problem you want to solve.

Model Training

This part can actually get a little more complex. My plan was to train multiple networks with varying architectures and parameters. So I wrote node/modelFactory.js, which encapsulates building and training of the model. This allows me to train multiple models with varying layers and learning rate.

The learning rate defines how quickly the model will learn. There are good defaults, but some models may work better with a higher LR and others with a much lower LR. This popular graphic shows what happens if the LR is too high or too low:

Before you dive into the modelFactory.js code, here are the essential parts of model training.

First, you may need to transform your data, which is true for this project. We built dictionaries using mimir, now we need to get the actual Bag of Words. This will return one-hot encoded arrays (e.g. [0, 1, 0, 0, 1]). The important part is, that these arrays are always of the same length. Every 1 corresponds to a certain word of the dictionary and means that the message that this BOW represents contains the corresponding word. Check out mimir's BOW code example.

const allData = [];
dataFlat.forEach((entry, idx) => {
  allData.push({
    input: mimir.bow(entry, dictData),
    output: mimir.bow(dataLabels[idx], dictLabels)
  });
});

Then, we use our current layer and lr parameters to build the model and train it with our training data.

        // create the net
        const net = new brain.NeuralNetwork({
          hiddenLayers: layers,
          learningRate: lr
        });

        // train the net
        const trainResult = net.train(traindata, {
          errorThresh: 0.005,
          iterations: 10000,
          log: true,
          logPeriod: 100
        });

        // test the net
        const testResult = testing(net, testdata, threshold);

        // write net to file
        fs.writeFileSync(
          `./out/nets/${fold + 1}_${netName}.json`,
          JSON.stringify(net.toJSON())
        );

This will then save a trained model, which is ready to use.

Model Evaluation

Did you see the testing(net, testdata, threshold) call in the above code snippet? This will test how our model is performing after it was trained.

It calculates the precision, recall and finally the F1 score. This is a common score to use, and especially useful in our binary classification project.

The implementation is pretty straight forward:

function(net, testData, threshold) {
  const maxarg = array => {
    return array.indexOf(Math.max.apply(Math, array));
  };

  let truePositives = 0;
  let trueNegatives = 0;
  let falsePositives = 0;
  let falseNegatives = 0;

  for (let i = 0; i < testData.length; i++) {
    const result = net.run(testData[i].input);
    if (result.show >= threshold && testData[i].output.show === 1) {
      truePositives++;
    } else {
      falseNegatives++;
    }
  }

  const precision = truePositives / (truePositives + falsePositives);
  const recall = truePositives / (truePositives + falseNegatives);
  const f1 = 2 * ((precision * recall) / (precision + recall));
  return { precision, recall, f1 };
};

We take the trained net, some test data and a manually set threshold. The threshold is the break where we want to classify something as "spam" or "no spam". A reasonable value would be 0.8, which means that if the model is 80% sure a message is "no spam", then we will classify it as "no spam", otherwise "spam".

This score is then used to evaluate how well a certain model performs. You can see that I tested lots of different model architectures in node/hyperparamTuning.js.

Use the Model to classify messages

This part is a breeze with brain.js. The model can be represented as a JSON string. The same thing is true for our words dictionary using mimir. So all we need to do is load the model and word dictionary as string from some backend.

All the ML code in the React web app is located at src/api/NeuralNet.js. Essentially, to predict whether a new message is spam or not, we just need to call:

  predict(message) {
    const maxarg = array => {
      return array.indexOf(Math.max.apply(Math, array));
    };

    if (typeof message !== 'string' || message.length < 1) {
      console.warn(`Invalid message for prediction: ${message}`);
      return 0;
    }

    if (!this.net || !this.dict || typeof this.net.run !== 'function') {
      console.error('Cant predict because: net | dict', this.net, this.dict);
      return 0;
    }

    const test_bow_message = mimir.bow(message, this.dict);
    const prediction = this.net.run(test_bow_message);
    return maxarg(prediction);
  }

This will ...

create the BOW representation of the incoming message
call .run() on the neural net to get a prediction, which is an array like this: [0.2, 0.8]
return the index of the prediction array that has the highest value to classify the message as either "spam" or "no spam"

And that's it! We successfully trained a model offline using NodeJS, saved the best performing model as JSON and used it in the web app to predict a new message.

I left out a couple of things that are part of the web app. These include creation of the dataset, live data collection and in-browser training. You can find all these features in the repo and test them out in the web app. Let me know, if you want another article going more in-depth on certain features.

Afterword

I am by no means an expert on this topic, but I dipped my toe in the Machine Learning waters and would like to encourage more (web)devs to try it as well. Hopefully, this article helped some of you to do just that!

Also thanks to Ben Halpern, who encouraged me to write this. I really like the dev.to community, you folks rock! ❤️

React or Angular for enterprise?

Martin Beierling-Mutz — Sun, 20 Jan 2019 16:32:21 +0000

I recently had a discussion with colleagues about React vs. Angular for enterprise applications. I'd like to bring this discussion to the dev.to community as well.

The core arguments were:

React comes with less built-in tools than Angular. Therefore, React setups need to complement vanilla React by installing open-source dependencies. This makes the on-boarding of new developers to a project built with React more difficult, as any unknown dependencies need to be learned first. Whereas Angular has necessary concepts and tools built-in, which Angular developers already know and therefore have a quicker on-boarding to a new project.
Angular uses Typescript by default and is built with Typescript in mind. React isn't. Certain patterns in React are hard or cumbersome to type correctly. As Typescript is generally preferred over JS in enterprise projects, Angular is the better choice.

🤔💭 What are your thoughts on this? 🤔💭

Overview of JSX components and prop usage with "jsx-info"

Martin Beierling-Mutz — Sun, 30 Dec 2018 18:32:40 +0000

Very cool tool by Brian Mock, which shows an overview of your component props and their usage in a codebase:

wavebeem / jsx-info

Displays a report of JSX component and prop usage

jsx-info

Displays a report of JSX component and prop usage.

Watch my demonstration video for more information.

Why

First of all, I thought it would be cool to see all this info. But more importantly, I think jsx-info can be used to help refactor your code.

Let's say you have a component called <DataTable> that takes 43 different props. If you needed to rewrite <DataTable> from scratch, you might not want to keep as many different props. Using jsx-info you could analyze which props get used the most and start porting that functionality first.

If the usage of a particular prop is very low, you might even choose to get rid of that prop and rewrite the calling code to use something else instead.

The intended workflow here is to run jsx-info and compare the data with your prop-types or TypeScript type definitions to find discrepencies.

Installation

Automatically install and…

View on GitHub

Let's say you have a component called <DataTable> that takes 43 different props. If you needed to rewrite <DataTable> from scratch, you might not want to keep as many different props. Using jsx-info you could analyze which props get used the most and start porting that functionality first.

Auto-Generate sitemap.xml in Next.js

Martin Beierling-Mutz — Sat, 29 Dec 2018 10:55:09 +0000

Hurray! You created all the components and styling for your beautiful and performant Next.js website. What now?

There are some key files you want to serve in the root of your exported package, but Next.js only supports copying files from the /static folder out of the box. But how to add e.g. a sitemap.xml, even in an automated and always up-to-date way?

Let me show you how you could set this up for a 'yarn export'ed Next.js project.

Basic sitemap.xml structure

First, we'll need to take a look at the info a basic sitemap needs to have.

A list of...

URLs to each available page,
and an accompanying date, to let the search engine bot know where to find a page and when it was last changed.

That's it! If you want more info, you can check out Google's "Build and submit a sitemap" site.

Gathering needed info

Before we can write the file into our exported /out folder, we'll have to actually get the info we need: page url's & last modified dates.

To do this, I've built this function, which returns all files' paths inside the /pages folder:

module.exports = () => {
  const fileObj = {};

  const walkSync = dir => {
    // Get all files of the current directory & iterate over them
    const files = fs.readdirSync(dir);
    files.forEach(file => {
      // Construct whole file-path & retrieve file's stats
      const filePath = `${dir}${file}`;
      const fileStat = fs.statSync(filePath);

      if (fileStat.isDirectory()) {
        // Recurse one folder deeper
        walkSync(`${filePath}/`);
      } else {
        // Construct this file's pathname excluding the "pages" folder & its extension
        const cleanFileName = filePath
          .substr(0, filePath.lastIndexOf("."))
          .replace("pages/", "");

        // Add this file to `fileObj`
        fileObj[`/${cleanFileName}`] = {
          page: `/${cleanFileName}`,
          lastModified: fileStat.mtime
        };
      }
    });
  };

  // Start recursion to fill `fileObj`
  walkSync("pages/");

  return fileObj;
};

This will return an object, which looks like this for my website at the moment of writing:

{
  "/blog/auto-generate-sitemap-in-next-js": {
    "page": "/blog/auto-generate-sitemap-in-next-js",
    "lastModified": "2018-10-03T00:25:30.806Z"
  },
  "/blog/website-and-blog-with-next-js": {
    "page": "/blog/website-and-blog-with-next-js",
    "lastModified": "2018-10-01T17:04:52.150Z"
  },
  "/blog": {
    "page": "/blog",
    "lastModified": "2018-10-03T00:26:02.134Z"
  },
  "/index": {
    "page": "/index",
    "lastModified": "2018-10-01T17:04:52.153Z"
  }
}

As you can see, we have all the info we need to build our sitemap!

Creating the file when exporting

In Next.js, when you create your static files package, you'll typically run yarn build && yarn export. We want to hook-in after the export, to create the sitemap.xml file in the /out folder.

To hook into any scripts defined the package.json, we can add another script with the same name, but prefixed with "post";

The new package.json scripts section will look like this:

...
"scripts": {
    "dev": "next",
    "build": "next build",
    "start": "next start",
    "export": "next export",
    "postexport": "node scripts/postExport.js"
  },
...

I chose to create a new folder "scripts" and create the "postExport.js" file in there. This script will now run after every "yarn export" call.

Generate the sitemap.xml contents

This scripts/postExport.js file will utilize the function we created previously to get all needed info:

const pathsObj = getPathsObject();

Then, we'll create the sitemap.xml content & file:

const sitemapXml = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> 
  ${Object.keys(pathsObj).map(
    path => `<url>
    <loc>https://embiem.me${path}</loc>
    <lastmod>${
      formatDate(new Date(pathsObj[path].lastModified))
    }</lastmod>
  </url>`
  )}
</urlset>`;

fs.writeFileSync("out/sitemap.xml", sitemapXml);

That's it! Well almost. I did use a formatDate function, to get the desired string format for our date.

You could just do a .substr(), as pathsObj[path].lastModified already contains an ISO formatted date, or use some library like date-fns. I decided to copy a working solution from the web:

module.exports = function formatDate(date) {
  var d = new Date(date),
    month = "" + (d.getMonth() + 1),
    day = "" + d.getDate(),
    year = d.getFullYear();

  if (month.length < 2) month = "0" + month;
  if (day.length < 2) day = "0" + day;

  return [year, month, day].join("-");
};

Now run yarn export and a file at out/sitemap.xml appears!

Challenge! Create robots.txt

Based on this, it should be easy for you to create a robots.txt with your desired contents now.

If you want to know how I did it, check out the scripts folder in my website's repo.

Let me know whether you'd solve this differently.

Afterword

This was my first post in this community 👋. I plan to post much more in the future. You can find the original post on my Blog.

DEV Community: Martin Beierling-Mutz

How to Consistently Retrieve Valid JSON from Claude 3.5 in Go

Why I fell in love with tailwindcss

The Struggle

The Realisation

The (Hidden) Benefits

Is It For You?

Use Machine Learning to filter messages in the browser

The Problem

What's Twitch?

The Solution

embiem / Better-Twitch-Chat

Hobby-project using Twitch Chat, ML, Twitch-Kraken API, reactJS & firebase

3rd party libraries

New Clone

Table

What's an ANN?

How does an ANN work?

Creating of neural network using JavaScript in 7 minutes!

Petro Liashchynskyi ・ Jan 12 '19

Data

Data Collection

Data Preprocessing

Data Exploration & Visualization

Model

Build your model

Model Training

Model Evaluation

Use the Model to classify messages

More to read & watch about ML

Afterword

React or Angular for enterprise?

Overview of JSX components and prop usage with "jsx-info"

wavebeem / jsx-info

Displays a report of JSX component and prop usage

jsx-info

Why

Installation

Auto-Generate sitemap.xml in Next.js

Basic sitemap.xml structure

Gathering needed info

Creating the file when exporting

Generate the sitemap.xml contents

Challenge! Create robots.txt

Afterword