These next changes came with some architecture changes, which were too intrusive.
https://gitlab.com/jessephillips/devarticlator/-/compare/v0.0.7...v0.0.8
- private Paths articlePaths(string folder, ArticleMeta am)
+ private const(Paths) articlePaths(string folder, const ArticleMeta am)
I marked method parameters with const and in the case of this method, my unittesting was triggering a warning because the call would produce no side effects.
auto suppressWarning = articlePaths("someLocation", meta);
By assigning to a variable the compiler was satisfied that I was planning to do something with the result.
Thinking more on this for the article, I should have returned immutable as the type is unique and now const, effectively making it immutable anyway.
https://gitlab.com/jessephillips/devarticlator/-/blob/v0.0.8/source/devtopull.d#L78
I've also implemented the initial interface for making the Dev api into a range. The first part was to remove the dependency of making a web request.
private const(ArticleMe)[] function(uint page) @safe articlePage;
I did this by using a function pointer which provides a list based on a page number. I can now use this for testing my range without the web request.
Originally I was considering the use of std.concurrency which would allow me to delegate to a different thread, but ultimately did not feel like threading was of benefit over the function pointer.
Inside my function I create a local variable which will hold the web request results. This is necessary because vibe.d use a delegate for its own processing, it is also why I thought the message passing would be helpful.
const(ArticleMe)[] ret;
requestHTTP(format!"https://dev.to/api/articles/me/all?page=%s"(page),
One thing I don't know is how many pages, guess 404 is how I will know to stop.
https://gitlab.com/jessephillips/devarticlator/commit/9506d04668a49d33afe37ef252018670dcb6a1bc
One thing I needed to change was to modify the save article to operate on a single element instead of a range. This wasn't hard because I already had implemented as a nested function. But I did forget to grab the in contract.
This all lead to:
alias inPreviousPull = (x) => !x.isNewerArticle(ps);
auto newestArt = devtoMyArticles()
.until!inPreviousPull
.structureArticles
.tee!(x => x.saveArticle(savePath))
.reduceToNewest;
Let's compare with my previous post guess at the implication.
devreqArticleRange
.until!inPreviousPull
.structureArticles
.saveArticles(savePath);
The primary change here is the driving operation is no longer saving the articles. Instead I would like to get the article which is the last one written so I know when to short circuit the next time. This is important because the api will produce unpublished first so I can't just take the first article. With that said I should modify isNewerArticle to ignore unpublished articles.
tee allows splitting the pipeline execution, this allows me to save each article while I'm finding the newest one.
Generally walking a range multiple times isn't a big deal. Large ranges or in this case where a range would have a larger wait on devices. Though I probably could have just as easily downloaded all the articles into an array, it seems unlikely that even 200 pages would have memory issues in today's computers.
I had chosen to start with pulling the articles to become a little familiar with the api and what my article structure might look like.
Clearly I've gone a little overboard. I could have exported the articles long ago. Instead I'm trying to be more robust and apply applicable attributes.
I can't say that my code was improved because of it though. The annotations were added only as a description of how the function already behaved. That being said, I think my coding style has been heavily influenced by these attributes as I have strived to be able to use them.
I did not write tests first, but obviously have some items to add coverage for.
Top comments (1)
I have found that the dev api will return 200 with no articles when reaching past the pages with content.