loading...
Cover image for What Did You Learn This Week --April 17?

What Did You Learn This Week --April 17?

waylonwalker profile image Waylon Walker ・1 min read

this-week-i-learned (13 Part Series)

1) What Did you Learn this week --March 6? 2) What Did you Learn this week --March 13? 3 ... 11 3) What Did you Learn this week --March 20? 4) What Did You Learn This Week --March 27? 5) What Did You Learn This Week --April 3? 6) What Did You Learn This Week --April 10? 7) What Did You Learn This Week --April 17? 8) What Did You Learn This Week --April 24? 9) What Did You Learn This Week --May 1? 10) What Did You Learn This Week --May 8? 11) What Did You Learn This Week --May 15? 12) What Did You Learn This Week --May 22? 13) What Did You Learn This Week --May 29?

Discussion

markdown guide
 

I learned a ton about pytest while building find-kedro

Most notably the built in fixture for tempdir, simply adding tempdir as an argument to your test function gives you a tempdir to create files and run tests against them.

GitHub logo WaylonWalker / find-kedro

Find Kedro is a small library to enhance your kedro experience by automatically assembling pipelines

Find Kedro Title

find-kedro is a small library to enhance your kedro experience. It looks through your modules to find kedro pipelines, nodes, and iterables (lists, sets, tuples) of nodes. It then assembles them into a dictionary of pipelines, each module will create a separate pipeline, and __default__ being a combination of all pipelines. This format is compatible with the kedro _create_pipelines format.

Motivation

kedro is a fantastic project that allows for super-fast prototyping of data pipelines, while yielding production-ready pipelines. find-kedro enhances this experience by adding a pytest like node/pipeline discovery eliminating the need to bubble up pipelines through modules.

When working on larger pipeline projects, it is advisable to break your project down into different sub-modules which requires knowledge of building python libraries, and knowing how to import each module correctly. While this is not too difficult, in some cases, it can trip up even the most senior engineers, losing precious…

 

I have learned many things whilst working on my submission for #twiliohackathon. One of them is the power of Gitlab over Github, I'm struggling to see any good reasons to use Github anymore!

In my project I've been able to create a full CI pipeline to three different environments, deploy based on branch and have source and dependency vulnerability checking in one 41 line .gitlab_ci file! 🎉

 

I got started with 11ty, and reworked my blog. All in a span of a week. I love the simplicity and build speed

 

🎉 Awesome work! 11ty seems really great. I started using gatsby a couple of years ago. Its great, but would love to have a faster build speed.