DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

Jesse Phillips
Jesse Phillips

Posted on

Hobby Project: Moving to a Range of Articles

These next changes came with some architecture changes, which were too intrusive.

https://gitlab.com/jessephillips/devarticlator/-/compare/v0.0.7...v0.0.8

- private Paths articlePaths(string folder, ArticleMeta am)
+ private const(Paths) articlePaths(string folder, const ArticleMeta am)
Enter fullscreen mode Exit fullscreen mode

I marked method parameters with const and in the case of this method, my unittesting was triggering a warning because the call would produce no side effects.

auto suppressWarning = articlePaths("someLocation", meta);
Enter fullscreen mode Exit fullscreen mode

By assigning to a variable the compiler was satisfied that I was planning to do something with the result.

Thinking more on this for the article, I should have returned immutable as the type is unique and now const, effectively making it immutable anyway.


https://gitlab.com/jessephillips/devarticlator/-/blob/v0.0.8/source/devtopull.d#L78

I've also implemented the initial interface for making the Dev api into a range. The first part was to remove the dependency of making a web request.

private const(ArticleMe)[] function(uint page) @safe articlePage;
Enter fullscreen mode Exit fullscreen mode

I did this by using a function pointer which provides a list based on a page number. I can now use this for testing my range without the web request.

Originally I was considering the use of std.concurrency which would allow me to delegate to a different thread, but ultimately did not feel like threading was of benefit over the function pointer.


Inside my function I create a local variable which will hold the web request results. This is necessary because vibe.d use a delegate for its own processing, it is also why I thought the message passing would be helpful.

const(ArticleMe)[] ret;

    requestHTTP(format!"https://dev.to/api/articles/me/all?page=%s"(page),
Enter fullscreen mode Exit fullscreen mode

One thing I don't know is how many pages, guess 404 is how I will know to stop.


https://gitlab.com/jessephillips/devarticlator/commit/9506d04668a49d33afe37ef252018670dcb6a1bc

One thing I needed to change was to modify the save article to operate on a single element instead of a range. This wasn't hard because I already had implemented as a nested function. But I did forget to grab the in contract.

This all lead to:


alias inPreviousPull = (x) => !x.isNewerArticle(ps);

auto newestArt = devtoMyArticles()
        .until!inPreviousPull
        .structureArticles
        .tee!(x => x.saveArticle(savePath))
        .reduceToNewest;
Enter fullscreen mode Exit fullscreen mode

Let's compare with my previous post guess at the implication.

devreqArticleRange
  .until!inPreviousPull
  .structureArticles
  .saveArticles(savePath);
Enter fullscreen mode Exit fullscreen mode

The primary change here is the driving operation is no longer saving the articles. Instead I would like to get the article which is the last one written so I know when to short circuit the next time. This is important because the api will produce unpublished first so I can't just take the first article. With that said I should modify isNewerArticle to ignore unpublished articles.

tee allows splitting the pipeline execution, this allows me to save each article while I'm finding the newest one.

Generally walking a range multiple times isn't a big deal. Large ranges or in this case where a range would have a larger wait on devices. Though I probably could have just as easily downloaded all the articles into an array, it seems unlikely that even 200 pages would have memory issues in today's computers.


I had chosen to start with pulling the articles to become a little familiar with the api and what my article structure might look like.

Clearly I've gone a little overboard. I could have exported the articles long ago. Instead I'm trying to be more robust and apply applicable attributes.

I can't say that my code was improved because of it though. The annotations were added only as a description of how the function already behaved. That being said, I think my coding style has been heavily influenced by these attributes as I have strived to be able to use them.


I did not write tests first, but obviously have some items to add coverage for.

Top comments (1)

Collapse
 
jessekphillips profile image
Jesse Phillips

I have found that the dev api will return 200 with no articles when reaching past the pages with content.

Create an Organization Does your company have something to share with DEV?

Create an Organization and start sharing content with the community on DEV.