loading...
Cover image for I just published my first Open Source library in Python and wanted to share it with you guys (video below)

I just published my first Open Source library in Python and wanted to share it with you guys (video below)

voyz profile image Voy ・1 min read


https://www.youtube.com/watch?v=8XL7d_ehhsA

Databay is a scheduled data transfer interface, facilitating transfer of data from A to B on a scheduled interval. If your project requires custom data transfer, it would make my day if you gave Databay a look.

You can find Databay on GitHub, or its documentation on ReadTheDocs.

Some of Databay’s features are:

  • Simple, decoupled interface — Easily implement data production and consumption that fits your needs.
  • Granular control over data transfer — Multiple ways of passing information between producers and consumers.
  • Asyncio supported — You can produce or consume asynchronously.
  • We'll handle the rest — scheduling, startup and shutdown, exception handling, logging.
  • Support for custom scheduling — Use your own scheduling logic if you like.

I’m looking for someone who would like to do some code review on it (it’s relatively small), so if you’d feel like reading some code and helping out - drop me a message. Thanks!

This is the first time I’m publishing an open source library so would love to hear your feedback.

Thanks!
Voy

Posted on by:

Discussion

markdown guide
 

Sounds interesting , Good Job 👏🏻

 

Thank you! Let me know if you have a chance to use it, would love to hear some feedback on it :)

 

To be honest I didn't have a use case for it yet. But if I do for sure I will open a issue on github.👍

Appreciate it Omar! Have a good one!

 
 

Will it be able to FTP. txt files to mainframes

 

I haven't tried doing this yet although at a glance I can't see why this wouldn't work. You're most welcome to give it a shot! Let me know how it goes, I'd be happy to help you build it.

You can implement it as follows:

  1. Create an Inlet that reads a directory's contents and produces records containing file paths to the .txt files
  2. Create an Outlet that expects records with file paths, picks the files specified and sends them over to your mainframe.
  3. Link the two nodes and schedule on the interval of your choice.

The basic_inlet and basic_outlet examples would be a good place to start if you'd like to see how inlets and outlets can be extended.

Keep me posted if you decide to try it out 😊