DEV Community: prumand

Opentelemetry for python

prumand — Thu, 28 Sep 2023 15:05:40 +0000

I have a cli-app (see my previous post) and I want to collect some usage stats. Of course opentelemetry is the way to go and python support is pretty far.

I’m using jaeger for my traces. I don’t need live observability, but just some analytics from time to time. So I’d prefer starting a container and ingesting traces which are stored on disk on demand.

Unfortunately there is no to-disk exporter for traces out of the box, but it’s easy to set up. First for writing to files, I’ll just use the standard python logging:

import logging
from logging import LogRecord, Logger
from logging.handlers import RotatingFileHandler


handler = RotatingFileHandler(LOG_FILE_PATH, maxBytes=1024 * 1024, backupCount=5)
logging.basicConfig(
    level=logging.INFO,
    handlers=[handler],
    format='%(message)s',   
)

It's ok to rotate logs from time to time. I can alway crank up the backupCount, if I think it’s important.

I’m using the standard otel-ConsoleExporter and change the formatter, since I want all traces to be written on one line. That makes it easier to ingest the file later. Further I change the output of the Exporter to my customer implementation.

class LogOut:
    def __init__(self, logger: Logger) -> None:
        self._logger = logger

    def write(self, msg):
        self._logger.info(msg)

    def flush(self):
        pass

provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(ConsoleSpanExporter(
    formatter=lambda span: span.to_json(None),
    out=LogOut(logging.getLogger('makit.cli'))
))

provider.add_span_processor(processor)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer('makit.cli')

So basically I just need to add the traces to my action-functions and I’ve all I need to know:

if __name__ == "__main__":
    with tracer.start_as_current_span("main") as span:
        get_app()()

Sidenote

For me it’s fine to use span events as logging. Unfortunately those don’t show up in the jaeger-ui. But I tried something like the following:

class OtelLogger(logging.Logger):
    def handle(self, record: LogRecord) -> None:
        span = trace.get_current_span()
        span.add_event(record.msg)

It gives me all the information I need and I thought about setting the span status based on the log-level.

Let's measure

prumand — Tue, 26 Sep 2023 10:31:12 +0000

I’m guilty of undermeasuring. With the next side project I want to change that, and see if I can do better. Basically I want to use my abilities to program, to help my everyday routines. It’s mainly about integrating all the great tools already known and finding a different approach to use them.

For now I want to leave out the details of the implementation, but to give you a glimpse of what the starting point looks like. I've already established a foundational setup that allows me to configure commands similar to the following example:

---
commands:
  - name: hello-world
    description: >
      Greet a planet
    executor:
      type: shell
      command: echo "Hello ${planet}!"
    parameters:
      - name: planet
        description: The planet you want to greet.
        type: string
        autocomplete:
          - Earth
          - Pluto

Building something like this is really fun, but you can easily get off track. I think the best way to guardrail my efforts is to define what I want to achieve and measure the outcomes. It might be difficult to find, but I’m sure it’s possible.

All important decision makers could benefit from learning that anything they really need to know is measurable. [How to measure anything : finding the value of “intangibles” in business. Douglas W. Hubbard. – 2nd ed.]

So let’s find out what we want to measure.

Is Usage Success?
I go with a definite maybe. First self deception is a thing, so an easy way to rate the commands used, will hopefully add quality as a dimension. Something cli rate-last [1-5], will help me capture my emotion in the moment, even if I only rate exceptional usage.
Secondly, success is more than just usage. I want to track the projects I use my app with and evaluate the success of the project (failing and learning is fine) and the impact of the app.
Integrating as focus
Third party software and services are so integral to my work and a major part of the vision. That’s why I want to track it besides the general success. The app might have integrations backed in, but it’s not its main purpose. So all software and services which are triggered, used or configured with my app, should be tracked.

I’m aiming at the following:

Reaching 200 calls per week of the app within the two months,
no more than 20/100 bad calls,
10 integrations

I’ll follow-up on how I collect the data.

From Domain to Tasks

prumand — Mon, 18 Sep 2023 16:45:02 +0000

Not long ago, I had a discussion with a colleague. I told him that if he can’t explain his planned work to his peers, he should probably work on his plan. Kind of paraphrasing Richard Feymann:

If you can't explain something to a first year student, then you haven't really understood it.
[https://en.wikiquote.org/wiki/Talk:Richard_Feynman]

The problem seems easy enough, but in this context the problem was also finding what to do. Kind of a “Full Stack Engineer” for a whole slice of a domain. Maybe “Full Domain Engineer”?

As a Full Domain Engineer getting your list of tasks is not merely structuring around ready made requirements, but finding those requirements in the first place. This needs a lot of communication. To put all that information into a form that can be understood and be worked with we require models = representations.

Event Storming helps with some definitions and ways to put them together. (see https://miro.com/miroverse/event-storming/ for a nice walk-through). But the transition from a model to a list of tasks is pretty hard. Some problems I had were the following:

Leaking technical decisions, especially past ones, into the model. For example some events which only occur because of a workaround.
Document decisions on the semantics of the model. Even though I feel like a wiki page should do, that failed me on follow ups.
Differentiating between the domain and technical things. That was especially hard, when my team was providing technical infrastructure. The domain is infrastructure, but the technical things are the infrastructure. So what is the domain?
Our Brains need time to process. Don't transition into task mode right after a modelling session.

Last but not least, I think the whole modeling session is to inform, not to create tasks. Finally we need tasks, but I always felt they are created way too early.

// Detect dark theme var iframe = document.getElementById('tweet-339435106961854464-214'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=339435106961854464&theme=dark" }

Let the Tests Design

prumand — Thu, 14 Sep 2023 06:37:30 +0000

I'm currently building myself a personal cli, which helps me integrate coded automation into my daily routines. After some experiments, I though that my next implementation could be here to stay. That's why I started with TDD and after some coding I ended up with the following test for the entrypoint of my typer-app.

@patch("main.get_config")
@patch("main.get_action_factory")
def test_create_action_with_factory(create_action_factory, get_config_mock):
    # Given
    action_factory_mock = MagicMock(spec=ActionFactory)
    create_action_factory.return_value = action_factory_mock
    config_service = ConfigService(action_path=DirectoryPath())
    get_config_mock.return_value = config_service

    # When
    result = runner.invoke(app, ["create-action", "new-action"])

    # Then
    result.exit_code == 0
    create_action_factory.assert_called_once_with(config_service.action_path)
    action_factory_mock.create_action.assert_called_once_with("new-action")

I thought it was a good to idea to let the shell entrypoint handle all the linking together, but there is a lot to mock. All this to answer whether the action_factory was called correctly.
So it's time to let the tests drive the design. I decided to pull out the service creation into a service factory.

After four commits I get the following code change

-    action_factory = get_action_factory(get_config().action_path)
+    action_factory = get_action_factory()

which allowed simpler test:

@patch("main.get_action_factory")
def test_create_action_with_factory(create_action_factory):
    # Given
    action_factory_mock = MagicMock(spec=ActionFactory)
    create_action_factory.return_value = action_factory_mock

    # When
    result = runner.invoke(app, ["create-action", "new-action"])

    # Then
    result.exit_code == 0
    action_factory_mock.create_action.assert_called_once_with("new-action")

Looks way better. I'll follow up with some more coding done. Let's see what other decisions the tests designed.

Save me from SAFe

prumand — Wed, 28 Apr 2021 19:27:19 +0000

My team introduced Scaled Agile Framework or SAFe and we already started the first iteration. A quick glance at their documentation reminded me of Martin Folwer’s “Agile Industrial Complex”.
I did some more research on what other people on think about it:

Mary Poppendieck

@mpoppendieck

I am often asked what I think about scaled agile frameworks. This slide does a good job of summing up my viewpoint, except for the last point. We need more research on how successful large companies coordinate the work of small autonomous teams to achieve larger goals. twitter.com/TotherAlistair…

15:37 PM - 11 Dec 2019

Alistair Cockburn @TotherAlistair
I'm really liking the writing on this slide. I hadn't thought of it that way, good to see. https://t.co/TgkkBjY1fU

Daniel Terhorst-North

@tastapod

“SAFe is just another word for nothing left to lose”

22:12 PM - 13 Oct 2018

See also Daniel Terhorst-North’s in praise of swarming.
Some one even compiled a list of comments on SAFe. Which lead to me video from Alistair Cockburn, which I sum up as follows:

Yeah, standardize vocabulary 👍,
but investing into collaboration and your ability to deliver could yield better results.

Finally I stumbled over the following quote from ThoughtWork's Technology Radar Vol. 24 sums, which matches my feeling:

We've come across organizations struggling with SAFe's over-standardized, phase-gated processes. Those processes create friction in the organizational structure and its operating model. It can also promote silos in the organization, preventing platforms from becoming real business capabilities enablers. The top-down control generates waste in the value stream and discourages engineering talent creativity, while limiting autonomy and experimentation in the teams

So why did we introduce SAFe?

We are a young team and we need strategic alignment with company goals. Direction and support is needed to transition our team to do the right things.

But we are only one team, so we for sure miss scale. Some tools might provide good input, but introducing them all at once deprives us from learning. We will not understand what caused positive effects and will be overwhelmed by too many new techniques.

But why does it feel so wrong?

For example I think, it's a good idea to have cycle and a prior week reserved for communication and planing. In my experience predictability is rated too high, but that's for a different article. Nevertheless predicting a long cycle (10 weeks) and forcing the team to predict 5 minor cycle (2 weeks) limits team autonomy, produces overhead and creates a constant environment of pressure. All this with little to gain, since the 10 weeks were already estimated. Maybe the team would have even chosen two weeks sprints.

We can only dedicate about 50% (or even less) for work on features. We have a lot of bugs and operational work. We need to clean up. I don't think that you should separate cleaning up your architecture from your business work. Not separating those, leads to a more aligned cleaning up of technical debt and to culture where quality is built-in, rather than an afterthought.

Introducing a Product Owner to take care of customer interactions and product vision, but leave the developers out of it, seems a bit strange to me. Especially since I experienced communicating to be the major part of software development. There is an ongoing discussion about the Product Owner role (see here), but SAFe's definition seems to match the negative aspects.

So stay safe from SAFe!?

I admit that I made up my mind early. Finding out that https://www.scaledagile.com overwrites the HTML copy event to add a copy right disclaimer, increased my resentment. Maybe this should not be part of the evaluation, but it's a weak indicator that it doesn't fit my view of software development.

I think the fundamentals (e.g. lean, agile, team topologies) used and quoted by SAFe are legit. Reading up on those helps a lot and all of them are aware of context. Having a standardized way to plump all tools together would be great, but we are just not there yet.