RASA - Testing unhappy paths

In the previous chapter, we have written two unhappy paths for our newsletter_form. Now let's test that those paths work as we want them to.

# tests/test_stories.yml
# ... previous contents ...
  - story: Chitchat during email
    steps:
      - user: |
          I want to subscribe
        intent: subscribe
      - action: newsletter_form
      - active_loop: newsletter_form
      - user: |
          How's the weather?
        intent: chitchat
      - action: utter_ask_continue
      - user: |
          Yes, I do
        intent: affirm
      - action: newsletter_form
      - active_loop: null
      - action: utter_subscribed

  - story: Chitchat during frequency
    steps:
      - user: |
          I want to subscribe
        intent: subscribe
      - action: newsletter_form
      - active_loop: newsletter_form
      - user: |
          my email is [[e-mail]](email)
        intent: inform_email
      - action: newsletter_form
      - user: |
          How's the weather?
        intent: chitchat
      - action: utter_ask_continue
      - user: |
          Yes, I do
        intent: affirm
      - action: newsletter_form
      - active_loop: null
      - action: utter_subscribed

If you run the tests, they will all pass.

But if we add another test, testing both interruption during email and interruption during frequency, we will see it fail.

We can see why it failed in results/failed_test_stories.yml.

# results/failed_test_stories.yml
version: "2.0"
stories:
- story: Newsletter form two interruptions (/var/folders/ng/z4dw_m9d1bz6m08vw7nlwwwh0000gq/T/tmpqlha244q/7bc3772cad9348a589e105ce91f29c74_test_stories.yml)
  steps:
  - intent: subscribe
  - action: newsletter_form
  - active_loop: newsletter_form
  - intent: chitchat
  - action: utter_ask_continue
  - intent: affirm
  - action: newsletter_form
  - slot_was_set:
    - email: [e-mail]
  - intent: chitchat
  - action: utter_ask_continue
  - intent: affirm
  - action: newsletter_form  # predicted: action_two_stage_fallback
  - active_loop: null
  - action: utter_subscribed

To fix this, start an interactive chat – rasa interactive.

Throughout the chat, I said what the correct actions shuld be. At the end of the chat, I chose to save this chat to data/stories.yml.
Let's inspect the story that has been created for us.

# data/stories.yml
# ... previous content ...
  - story: interactive_story_1
    steps:
      - intent: subscribe
      - action: newsletter_form
      - active_loop: newsletter_form
      - slot_was_set:
          - requested_slot: email
      - intent: chitchat
      - action: utter_ask_continue
      - intent: affirm
      - action: newsletter_form
      - slot_was_set:
          - requested_slot: email
      - slot_was_set:
          - email: [e-mail]
      - slot_was_set:
          - email: [e-mail]
      - slot_was_set:
          - requested_slot: frequency
      - intent: chitchat
      - action: utter_ask_continue
      - intent: affirm
      - action: newsletter_form
      - slot_was_set:
          - requested_slot: frequency
      - slot_was_set:
          - frequency: twice a week
      - slot_was_set:
          - requested_slot: null
      - active_loop: null
      - action: utter_subscribed

Now let's look at new kind of steps that we have not been working with before.

This step is saying, that email is currently being requested:

      - slot_was_set:
          - requested_slot: email

This step is saying, that email has been filled with value [e-mail]. We do not need this step for our test to pass.

      - slot_was_set:
          - email: [e-mail]

This step is saying, that frequency has been filled with value twice a week. We do not need this step for our test to pass either.

      - slot_was_set:
          - frequency: twice a week

This step means, that no more slots are being requested.

      - slot_was_set:
          - requested_slot: null

You can learn more about requested_slot in my next article or in the documentation.

After droping the unnecessary steps, we get a bit simpler story.

# data/stories.yml
# ... previous content ...
  - story: interactive_story_1
    steps:
      - intent: subscribe
      - action: newsletter_form
      - active_loop: newsletter_form
      - slot_was_set:
          - requested_slot: email
      - intent: chitchat
      - action: utter_ask_continue
      - intent: affirm
      - action: newsletter_form
      - slot_was_set:
          - requested_slot: email
      - slot_was_set:
          - requested_slot: frequency
      - intent: chitchat
      - action: utter_ask_continue
      - intent: affirm
      - action: newsletter_form
      - slot_was_set:
          - requested_slot: frequency
      - active_loop: null
      - action: utter_subscribed

I also reverted changes that have been made to domain.yml and data/nlu.yml and kept only the generated story.

If we retrain the model and run tests again, they will all pass now.

You can learn more about testing unhappy paths in the documentation.

In the next chapter, we will look at how to communicate with our chatbot through REST API.

Repository for this tutorial:

petr7555 / rasa-dev-tutorial

You can checkout the state of the repository at the end of this tutorial by running:

git clone --branch 06-testing-unhappy-paths [e-mail]:petr7555/rasa-dev-tutorial.git

Comments

[Co-brainers-admin]

Hey Petr,

thanks for sharing your rasa testing know how, it is really appreciated.

I am also around to set up regression testing for our rasa implementations. I have really to admit that we are really struggling to get reliable results. I addressed this also in the rasa forum, but no feedback at all from the community.

My main issue: I get failed stories reported without an # marker.

-> How to interprete this ?

Here is what I wrote into the forum:

forum.rasa.com/t/end-to-end-story-...

Best

Andre

[Petr Janik]

Hi Andre,
I'd suggest including how your tests look and what the results of the tests are in the forum. Maybe someone will be able to help then.