Hi there -- I know it's been a while. A lot of things have changed and a lot of things have remained the same. I felt inspired to write today for a couple of reasons:
- I want to talk about screwing up because I screwed up yesterday and rather than let the shame beat me up, I want to transform these feelings into humility and maybe a lesson you can learn from or relate to
- I want to find a way to share my talk because it's a good talk and I am proud of it and you might find some of it useful
Speaking at PyBay
PyBay is one of my favorite conferences. I've spoken at PyBay three times, and each time, I've grown closer to the community. The Python community in the Bay Area is one of the most intelligent, most approachable, and most generous groups of people I've ever had the honor to know. Last year my talk was accepted as an alternate (that I ended up giving) and this year I was accepted as a primary speaker. I was thrilled. I was thrilled to give my talk and be in community. I had proposed an expanded version of a talk I'd given before at smaller meetups, so I was excited to iterate on my content and dive deeper into the subject of Pydantic.
Sometimes you just mess up
I have a lot going on right now. I don't know how it happened, but in addition to professional commitments, I've found myself in leadership positions with my extracurriculars. Art building, community organizing, possibly adopting a new dog ... everything has landed within the months of October and November.
But I had already given versions of the talk in June, so I wasn't making anything from scratch. I've spoken at PyBay before, so everything was familiar. I built out my slide deck, created a Jupyter notebook of code examples, and rehearsed, rehearsed, rehearsed. I curated an immaculate checklist of items to bring with me developed from years of conference experience, wrote out a detailed Trello card with addresses, directions, and attachments. I was ready to be at the venue an hour before my talk so I could settle in and catch one of the other speakers. I was the most prepared I have ever been for a talk ...
... except for one email that I somehow missed in all my checks and rechecks. An email reminding speakers to check in two hours prior to our scheduled talk times or else our slot would be given to an alternate. I arrived anxious but excited and proceeded to the speaker check-in table.
"Oh, we gave your slot to an alternate. You were supposed to be here at 1:30. There was an email."
My heart dropped. "But I ... "
I scrolled through my email. And sure enough, the subject line bolded to indicate it was unread, was the message from three days prior. I read it there, at the speaker check in table, the heat of shame and embarrassment rising up in me.
"I'm sorry -- we've just had issues in the past and we wanted to address that."
"No, no, I get it," I managed to utter. "I understand."
Fortunately, I was not alone. A friend was with me. We walked outside so I could take a breath. The conference organizer noticed me and rushed over, hugging me.
"I'm so sorry," she said. "We tried to call you. We know this isn't like you. We were worried something had happened. And we needed to implement this policy and it wouldn't be fair otherwise." Her eyes were sincere.
I checked my phone again and saw the missed call. My phone had never rang -- we had been crossing the Bay Bridge into San Francisco, where signal can be unreliable.
I've organized events before. Executing an event like PyBay requires precise coordination, contingency plans, and difficult decisions. Last year, I was accepted as an alternate and after a speaker had to drop, I got the opportunity to give my talk.
"That makes sense," I said, meaning it, trying to balance the overwhelming swell of my disappointment against the warm depth of my empathy and understanding against the hot heat of my shame and regret. "I understand."
She offered some more comforting words and hugged me again, and made sure I got a conference t-shirt. There's a reason why she's such a beloved member of the community.
Failing gracefully
In engineering, we talk a lot about graceful failures -- how to handle stuff that goes wrong without breaking the system. I wanted to leave. I wanted to hide in the shadows of shame and regret and be alone with my rumination, pointlessly saying to myself, "If only I had just done this, if only I had just done that." Replay the events over and over again, the exquisite self-flagellation of trauma-induced perfectionism.
But I knew that wasn't the right nor the helpful thing to do. I knew the right thing would be to show up for my community the way the community has shown up for me, and to see the conference through to the end. So I stayed. I was free to attend a talk I was interested in that originally conflicted with mine; the conference organizers had wisely accepted alternate talks. My mistake was ultimately handled gracefully. That's the sign of a well-architected system.
What I learned about myself
Besides the obvious -- which is to not only double check but triple check times -- I learned how much I love giving talks. I feel a certain comfort on stage or in front of an audience, and I find a lot of satisfaction in learning, distilling, and sharing information. I love when someone approaches me afterwards and says, "Your talk taught me something new." I didn't realize how much I had been looking forward to giving my talk until I didn't get to do it.
I also learned that my investment in my community doesn't go unnoticed, and that kindness, understanding, and loyalty are a positive feedback loop. The more you give, the more you get in return. No one was mad at me, I am welcome to continue participating in these events, and I was even asked to sign a thank you card for the organizer. "We love you, Liz," he said when I asked if he was sure he wanted to include my name.
I am still upset about it, I am still ruminating on all the things I should have done that I didn't do. I am not not beating myself up about it. The day after hasn't been easy.
I am trying to remember the understanding and forgiveness I was shown -- and I am trying to learn to grant myself the same kind of compassion. I hope the next time you screw up (because you will screw up), you remember this post, and it's okay if you can't quite muster up the compassion and grace, but I sincerely hope you will try.
The talk that could have been: Even more about Pydantic
The talk extends this blog post, which was also repurposed as a video. You can check out the code here, and the code for the extended talk here.
Using those resources as your foundation, here's even more about Pydantic.
Pydantic under the hood
At a high level, Pydantic follows the principle of "don’t just validate, parse it" –- an idea made popular by software engineer Alexis King.
The distinction is whether an input value is already the required type or whether it can be converted into that type. So instead of simply asking, "Is this input already an integer?" (True or False), Pydantic then asks, "Can this input be converted into an integer?" If the input cannot be parsed, then Pydantic raises a ValidationError. (Of course, this is in lax mode, which is Pydantic's default for type coercion.)
However, all of this parsing and validation can involve substantial work when processing large or deeply nested data. Pydantic uses pydantic-core –- a high-performance validation and serialization engine implemented in Rust –- to handle this.
So on the top layer you've got Pydantic, which provides:
- Python-facing API
- Model definition
- A pleasant developer experience
And below that, you've got pydantic-core, which gives you:
- Schema execution engine
- Validation and serialization
- And that Rust implementation which improves performance because it runs as compiled native code, which is “closer to the machine”
In this example, we see how the input value "14" (a string) is converted to an integer according to the schema we've defined in our Pug class:
from pydantic import BaseModel, Field
class Pug(BaseModel):
name: str = Field(..., min_length=2, description="The pug's name")
age: int
color: str = "fawn"
def bark(self):
print(f"The pug {self.name} goes arf arf!")
name = "Matty"
age = "14"
print(f"type of `name`: {type(name)}")
print(f"type of `age`: {type(age)}")
print("Creating a new Pug ...")
new_pug = Pug(name=name, age=age)
print(f"type of `name`: {type(new_pug.name)}")
print(f"type of `age`: {type(new_pug.age)}")
print(f"Here's your new_pug {new_pug}")
This generates the following output where we can see that the input type for age is a str. When we check the type of the value for age after creating a Pug, we see that the type is int:
type of `name`: <class 'str'>
type of `age`: <class 'str'>
Creating a new Pug ...
type of `name`: <class 'str'>
type of `age`: <class 'int'>
Here's your new_pug name='Matty' age=14 color='fawn'
When we create a Pug class that inherits from BaseModel, Pydantic generates a core schema describing the model's structure and behavior.
This schema is represented as a nested dictionary and is used by pydantic-core for validation and serialization. We can see that in the following example when we step into the __pydantic_core_schema__ attribute of Pug and check its type:
pprint(Pug.__pydantic_core_schema__)
print(type(Pug.__pydantic_core_schema__))
This is the output generated:
{'cls': <class '__main__.Pug'>,
'config': {'title': 'Pug'},
'custom_init': False,
'metadata': {'pydantic_js_functions': [<bound method BaseModel.__get_pydantic_json_schema__ of <class '__main__.Pug'>>]},
'ref': '__main__.Pug:4336583408',
'root_model': False,
'schema': {'computed_fields': [],
'fields': {'age': {'metadata': {},
'schema': {'type': 'int'},
'type': 'model-field'},
'color': {'metadata': {},
'schema': {'default': 'fawn',
'schema': {'type': 'str'},
'type': 'default'},
'type': 'model-field'},
'name': {'metadata': {'pydantic_js_updates': {'description': 'The '
"pug's "
'name'}},
'schema': {'min_length': 2, 'type': 'str'},
'type': 'model-field'}},
'model_name': 'Pug',
'type': 'model-fields'},
'type': 'model'}
<class 'dict'>
When we step into __pydantic_validator__ and __pydantic_serializer__, we see that they are built on Pydantic's pydantic-core machinery:
print(type(Pug.__pydantic_validator__))
print(type(Pug.__pydantic_serializer__))
And the output is here:
<class 'pydantic_core._pydantic_core.SchemaValidator'>
<class 'pydantic_core._pydantic_core.SchemaSerializer'>
In this example, we explicitly invoke the Pydantic validation machinery with validate_python():
name = "Matty"
age = "14"
print("Creating a new Pug using familiar Python object instantiation syntax ...")
new_pug = Pug(name=name, age=age)
print(f"Here's your new_pug {new_pug}")
print("Creating a new Pug twin using the Pydantic method `validate_python` ...")
data = {"name": name, "age": age}
new_pug_twin = Pug.__pydantic_validator__.validate_python(data)
print(f"Here's your new_pug_twin {new_pug_twin}")
In the output, we get the same result:
Creating a new Pug using familiar Python object instantiation syntax ...
Here's your new_pug name='Matty' age=14 color='fawn'
Creating a new Pug twin using the Pydantic method `validate_python` ...
Here's your new_pug_twin name='Matty' age=14 color='fawn'
What's important to note here is how we create a Pug with familiar Python syntax, but by inheriting from BaseModel, we are actually using Pydantic to validate our input data and create an instance of Pug.
As mentioned, Pydantic also generates structured errors as opposed to obtuse stack traces. In this example, we trigger a ValidationError by providing input that cannot be parsed and does not meet the constraints we defined:
name = "J"
age = "fourteen"
malformed_pug = Pug(name=name, age=age)
As a result, we get a clear ValidationError with explicit messaging:
ValidationError: 2 validation errors for Pug
name
String should have at least 2 characters [type=string_too_short, input_value='J', input_type=str]
For further information visit https://errors.pydantic.dev/2.13/v/string_too_short
age
Input should be a valid integer, unable to parse string as an integer [type=int_parsing, input_value='fourteen', input_type=str]
For further information visit https://errors.pydantic.dev/2.13/v/int_parsing
If we step into ValidationError we can see the error details generated by pydantic-core, rather than leaving us with an unstructured traceback:
try:
Pug(name=name, age=age)
except ValidationError as exc:
pprint(exc.errors())
[{'ctx': {'min_length': 2},
'input': 'J',
'loc': ('name',),
'msg': 'String should have at least 2 characters',
'type': 'string_too_short',
'url': 'https://errors.pydantic.dev/2.13/v/string_too_short'},
{'input': 'fourteen',
'loc': ('age',),
'msg': 'Input should be a valid integer, unable to parse string as an '
'integer',
'type': 'int_parsing',
'url': 'https://errors.pydantic.dev/2.13/v/int_parsing'}]
Real world Pydantic: The Vonage Python SDK
The following examples are from the blog post, so you can read more about them there. We'll look at the significant parts.
This demo code uses Vonage Verify to create a sample email OTP authentication workflow application. You can sign up for free and try it out. (All links and steps for creating a Vonage account and application are in the README.)
The Vonage Python SDk got an overhaul in 2024 and one of the big updates is its use of Pydantic for requests and responses.
This is what the VerifyRequest object looks like. Notice how it extends the Pydantic BaseModel. The @model_validator(mode='after') decorator is also provided by Pydantic for additional input validation:
class VerifyRequest(BaseModel):
"""Request object for a verification request."""
brand: str = Field(..., min_length=1, max_length=16)
workflow: list[
Union[
SilentAuthChannel,
SmsChannel,
WhatsappChannel,
VoiceChannel,
EmailChannel,
]
]
locale: Optional[Locale] = None
channel_timeout: Optional[int] = Field(None, ge=15, le=900)
client_ref: Optional[str] = Field(None, min_length=1, max_length=16)
code_length: Optional[int] = Field(None, ge=4, le=10)
code: Optional[str] = Field(None, pattern=r'^[a-zA-Z0-9]{4,10}$')
@model_validator(mode='after')
def check_silent_auth_first_if_present(self):
if len(self.workflow) > 1:
for i in range(1, len(self.workflow)):
if isinstance(self.workflow[i], SilentAuthChannel):
raise VerifyError(
'If using Silent Authentication, it must be the first channel in the "workflow" list.'
)
return self
In order to see how Pydantic can serve us in the real world, let's look at this demonstration comparing the use of Pydantic to create and validate a Verify API request versus manually creating the request. It does so using unittest to see what happens when we provide invalid input data:
class TestMain(unittest.TestCase):
test_data = {
"brand": 12345,
"to_email": 678910,
"channel_timeout": "sixty",
"code_length": "five",
}
def test_without_pydantic(self):
expected_result = 202
test_result = without_pydantic(self.test_data)
self.assertEqual(
test_result.status_code,
expected_result,
msg=f"Test without Pydantic failed with: {test_result.status_code}. Expected: {expected_result}",
)
def test_with_pydantic(self):
expected_result = 202
test_result = with_pydantic(self.test_data)
self.assertEqual(
test_result.status_code,
expected_result,
msg=f"Test with Pydantic failed with: {test_result.status_code}. Expected: {expected_result}",
)
Please note that these tests don't use any mocks, so running them makes real API calls which may incur costs.
Running the tests produces the following outcome:
test_with_pydantic (tests.test_main.TestMain.test_with_pydantic) ... ERROR
test_without_pydantic (tests.test_main.TestMain.test_without_pydantic) ... FAIL
What's important to note here is that the test using Pydantic errors out as opposed to failing. This is because Pydantic catches the incorrect request body parameters when the VerifyRequest object is created:
pydantic_core._pydantic_core.ValidationError: 1 validation error for EmailChannel
to
Input should be a valid string [type=string_type, input_value=678910, input_type=int]
For further information visit https://errors.pydantic.dev/2.13/v/string_type
The test without Pydantic fails because the result is a 422 and not a 202 which is the API response for invalid parameters:
AssertionError: 422 != 202 : Test without Pydantic failed with: 422. Expected: 202
This indicates that the Verify endpoint was called despite invalid parameters -- something that Pydantic caught before making a call to the API. This demonstrates how Pydantic validates models at creation and can help prevent wasted API calls.
In summary
In this version of the talk, we go beyond how Pydantic builds on type annotation while maintaining the flexibility of duck-typing to check the validity of any input data before executing functions on it. We take a look under the hood to see how the Rust-based pydantic-core engine handles schema generation, validation, and serialization without straying from the developer-friendly experience of using a human-readable language like Python. This allows us to understand how Pydantic is ideal for implementing APIs like Vonage Verify by validating input data before any request is made.
Seeking validation
I titled this talk "Seeking validation" as a pun. Writing and giving talks about technical concepts is how I process and integrate new knowledge because in order to teach, you really have to understand your subject. In other words, this process helps me validate my understanding.
I guess the pun is even funnier and more applicable now. I didn't validate my conference call time and, as a result, my talk didn't happen. I guess that's actually a really meta demonstration of Pydantic.
When I think about it that way, maybe I didn't screw up after all.
Top comments (0)