👋 Hey there, I am Waylon Walker
I am a Husband, Father of two beautiful children, Senior Python Developer currently working in the Data Engineering platform space. I am a continuous learner, and sha
I learned a ton about pytest while building find-kedro
Most notably the built in fixture for tempdir, simply adding tempdir as an argument to your test function gives you a tempdir to create files and run tests against them.
Find Kedro is a small library to enhance your kedro experience by automatically assembling pipelines
find-kedro is a small library to enhance your kedro experience. It looks through your modules to find kedro pipelines, nodes, and iterables (lists, sets, tuples) of nodes. It then assembles them into a dictionary of pipelines, each module will create a separate pipeline, and __default__ being a combination of all pipelines. This format is compatible with the kedro _create_pipelines format.
kedro is a ✨ fantastic project that allows for super-fast prototyping of data pipelines, while yielding production-ready pipelines. find-kedro enhances this experience by adding a pytest like node/pipeline discovery eliminating the need to bubble up pipelines through modules.
When working on larger pipeline projects, it is advisable to break your project down into different sub-modules which requires knowledge of building python libraries, and knowing how to import each module correctly. While this is not too difficult, in some cases, it can trip up even the most senior engineers, losing precious…
I learned a ton about pytest while building
find-kedroMost notably the built in fixture for
tempdir, simply adding tempdir as an argument to your test function gives you a tempdir to create files and run tests against them.Find Kedro is a small library to enhance your kedro experience by automatically assembling pipelines
find-kedrois a small library to enhance your kedro experience. It looks through your modules to find kedro pipelines, nodes, and iterables (lists, sets, tuples) of nodes. It then assembles them into a dictionary of pipelines, each module will create a separate pipeline, and__default__being a combination of all pipelines. This format is compatible with the kedro_create_pipelinesformat.kedrois afind-kedroenhances this experience by adding a pytest like node/pipeline discovery eliminating the need to bubble up pipelines through modules.When working on larger pipeline projects, it is advisable to break your project down into different sub-modules which requires knowledge of building python libraries, and knowing how to import each module correctly. While this is not too difficult, in some cases, it can trip up even the most senior engineers, losing precious…