DEV Community: William Narmontas

Web standards for content syndication

William Narmontas — Sun, 21 May 2017 00:43:46 +0000

https://www.scalawilliam.com/web-standards-for-content-syndication/

I've been making websites since 2004 and the one thing that struck me is how brittle it is to synchronise content over HTTP and also how to do it instantaneously. For example, when I write a blog post I'd like it to be synchronised to my own website, perhaps posted to Twitter or get a trackback/pingback - many different targets with many different intentions.

Perhaps it's not a blog post but an automatically generated item. It may be urgent or non-urgent, it may be frequent or infrequent.

In this article, I'm only interested in one-way data sharing, so WebSockets is out of the picture.

Pull

Pulling by polling means initiating a request to a server every X minutes and then checking if the content has changed.

This introduces a maximum delay and requires setting up scheduled tasks. Also you want to make sure not to exceed usage limits of the target server.

Atom and RSS are good for providing a structured pull approach.

Push

Pushing means that as soon as your content is published, YOU call into different targets to notify them of what has happened. It's reactive.

There are three flavours of push. If you want to be receiving the pushes, you need to host a dynamic server which is a disadvantage especially if you use static hosting.

Predefined set of targets

If you wanted to tweet automatically, at the point of publishing, you'd call into the Twitter silo API, but now you have to customise it for every single target. This typically happens with an HTTP POST.

Self-managing subscribers

One interesting use case I have is in ActionFPS where I wish to automatically enter tournament data into Challonge when a "clan war" is played. One way for me to do it is by calling into Challonge directly (current, but brittle), and the other way is to create a separate service (akin to microservices) to react to these events.

But I wish other people to be able to react as well. So I'm going to try out WebSub which is a W3 Candidate Recommendation from 11 April 2017, previously known as PubSubHubbub.

This is what we should be doing in microservices; not tightly coupled approaches.

Content-based targets

... such as mentions. There's a new standard out there called Webmention, which is similar to the good old xml-rpc, pingback and trackback.

Webmention enables conversations. Here's the specification: https://www.w3.org/TR/webmention/ - which is a W3C Recommendation as of 12 January 2017.

Effectively, if you want to receive webmentions, you place a <link rel="webmention"/> in your HTML and when someone mentions your page, they will call that endpoint.

Because of this approach, you can use free services like Bridgy (open source) to automatically post your content to Twitter and other places, and also syndicate responses back! You don't need centralisation or the "let's write our own Twitter adapter" approach above. Nice!

Pull-push

This is my favourite approach: EventSource (aka Server-Sent-Events). The W3 Draft is here: https://www.w3.org/TR/2011/WD-eventsource-20111020/ - it's also reactive.

EventSource is available in most browsers except IE - but you can use a polyfill for IE.

EventSource has a good Node.js library and also comes with reliability included - if you missed out an event then you can recover from the last event you saw. I made a small contribution to that library several weeks ago so that I could synchronise logs from ActionFPS to my local machine in real time. There's also a Scala client for SSE.

Advantage is that your receiver is a client and not a server, but a disadvantage is that it's a continuous connection.

Push API

Made for web browsers. Push API: W3C Working Draft 08 May 2017. Uses service workers.

Summary

Basically, decide to use the approach purely based on the frequency and urgency of delivery of your content:

Infrequent & not urgent, no server to receive - pull
Infrequent & urgent, with server to receive - push
Urgent, browser-only - Push API
Frequent, no server to receive - pull-push

Unit testing is simple

William Narmontas — Thu, 04 May 2017 06:56:56 +0000

https://www.scalawilliam.com/unit-testing-is-simple/

Is unit testing difficult?

No, it is not. You don't even need a testing framework for it.
You don't even need to write the test in a separate file or class.

You just need to exit with code 1. This is enough to fail a Jenkins build and a Makefile!

Why is a unit test useful?

Prevent regressions when you change the code.
Document expected input and output as code.
Replace error-prone println-driven development with test-driven development.

What do you test? when the logic is "complex enough" I know it could be broken 6 months down the line, due to "that one small change" - or that I'd spend time building it with print statements. Example from my own code.

Here, I'll show you the most basic way to unit testing with four languages: Bash, Python, Node.js, Scala, just to prove the point.

These examples are for a very basic piece of code and are intentionally minimalistic, to demonstrate that you needn't use a test framework to get started with - though one will be very necessary when you scale your application.

Bash

Here's a bash script inc.sh with a basic assertion and some main code:

#!/bin/bash

increment() {
    echo $(("$1"+1))
}

# If this assertion fails, the whole thing quits 
[ 3 -eq $(increment 2) ] || exit 1

# Increment a stream of numbers
while read n; do
    increment $n;
done

If we run:

$ seq 1 5 | bash inc.sh
2
3
4
5
6
$ echo $?
0

Exit code is 0. But if we broke the function, and put 2 instead of 1, we get:

$ seq 1 5 | bash inc.sh
$ echo $?
1

Python

This is easy to achieve in any language really. Python assertions:

Broken test in file inc.py:

def increment(n):
  return n + 2

assert increment(2) == 3

$ python inc.py
Traceback (most recent call last):
  File "inc.py", line 4, in <module>
    assert increment(2) == 3
AssertionError
$ echo $?
1

Node.js

Comes with assertions built in.

Broken test in file inc.js:

const assert = require('assert');

function increment(n) {
  return n + 2;
}

assert.equal(2, increment(1));

$ node inc.js

assert.js:81
  throw new assert.AssertionError({
  ^
AssertionError: 2 == 3

...
$ echo $?
1

Scala

Comes with assert.

def inc(n: Int) = n + 2
assert(inc(1) == 2)

$ scala inc.scala
java.lang.AssertionError: assertion failed
$ echo $?
1

Why is this important?

You can get started light with several assertions and work your way up to proper test suites.

So what are test frameworks for?

To make testing more organised. You get:

Organisation
Detailed error messages
Detailed test reports
Neat assertions and human readable language (see ScalaTest)
Incremental/TDD type of development

Conclusion

Now you have no excuse for not writing at least a few tests in your code.

Certifications I'm considering, as an experienced software engineer

William Narmontas — Sat, 29 Apr 2017 02:10:29 +0000

In building more and more open source projects spanning the full project lifecycle, I am appreciating the importance of structured business requirement analysis, architecture, software testing and project management.

I am already more than an engineer, with experience in all of these things but as a secondary concern without any formal knowledge. But it's precisely the secondary skill-sets that need some formal backing, and not the primary ones - because there's already a mountain of evidence of my skill in the primary ones.

I want my secondary skills to enhance my primary skills, with formal and certified method. In some environments, such formal badges are a big plus. Why?

For example, the team that I work with in Singapore (a country where authority really matters) is made up of PhDs who appreciate both formal knowledge and experience. In Scala engineering I have a lot of experience so I've no problem arguing my case, whereas in my secondary skills I have less focused experience and no formal backing. In discussions, I myself am not sure whether I'm putting the right arguments. Authority is important, and when backed by both by formal method and experience, is very impactful.

Thanks to my coworkers I've learned to prepare for arguments and became better at answering the "Why?". This prepares me well for the future of developing excellent products for myself and others.

Requirements

No training courses required, can self-study, whenever I want and for how long I want.
No work experience of that specific field required, with minimal bureaucracy.
Can be taken at test centres at around the world, like Pearson VUE.
Membership not required and renewals have a reasonable cost.
Recognised enough but does not have to be popular.
Entry level is enough, since they are only secondary skills.

But why not take it all the way? Well, law of diminishing returns. From experience, when people get things wrong, most of the time it is at the basic level. Like instead of acceptance criteria or tests first, they start with writing the code. Basic level is more than enough in probably a good 90% of the cases.

Certifications I found

Here are some certifications I found that match the criteria.

Business Analysis

I found the following sites:

From this,

I also found:

QAI's Certified Associate Business Analyst: (CABA)

Software testing

8 Best Software Testing Certifications Based on Your Experience Level

From this,

Architecture

From this,

SOA School

Other

PMI Agile Certified Practitioner (PMI-ACP)Â® - but has some practice requirement
AWS Certified Developer - Associate
Oracle - Java SE 8 Programmer I
Certified Professional Technical Communicator from Society for Technical Communication - interesting because it covers planning, analysis and the like.

What I'll choose

Still thinking about this. I notice they're still quite niche, and are more like components of the bigger picture, which is delivering products to users. Perhaps I need to look at something closer to the user, to product ownership and product development.

2017 freelance development platforms

William Narmontas — Thu, 20 Apr 2017 02:25:46 +0000

In designing Git Work, I must analyse other relevant services.

I'm publishing this article as a draft so it's out there and I can get feedback from you guys, perhaps I missed something?

Codemill

"Marketplace for pull requests", so focused on pull requests
Integrated with GitHub
Requires sign in to see the available work
"Pull work" approach, the Worker is in control
Nothing about disputes
Has an alerting system
Charge per piece of work
10% commission
"[..] elastic, scalable, on demand and pay-as-you-go development power [..]" source.

BountySource

"Bountysource is the funding platform for open-source software"
No support for private projects
Unit of work is an Issue, with a link to a PR
Bounties offered by users
Bounties potentially from multiple users
Not ties to a specific workflow
Comprehensive integrations including GitHub, and it's open source
10% commission as of 20 April 2017.

Upwork

"Work with someone perfect for your team"
Unit of work is a project
Bidding and proposal process (write a cover letter!)
Reputation and reviews
Work is mostly large, per-project
Freelancers offer free consultations during bidding
Act as Escrow, has complex dispute resolution
Hourly charge, estimate number of hours
2.75% + 20%/10%/5% commission as of 20 April 2017.

Toptal

"Hire the top 3% of freelance talent"
Hiring is based on passing their tests and interviews
"interview [..] 40 hours of your time" source.
$500 deposit
Charged per hour/week/etc, looks per-project
Unclear how work is agreed upon
Dispute process is unclear

LinkedIn ProFinder

"ProFinder helps you hire top local freelancers"
Proposal driven process, similar to Toptal
Pricing: after 10 proposals, requires LinkedIn Premium Business account
Beyond software development, they mention accountants for example

Usage

Some people use these services as a springboard.

DEV Community: William Narmontas

Web standards for content syndication

Pull

Push

Predefined set of targets

Self-managing subscribers

Content-based targets

Pull-push

Push API

Further reading

Summary

Unit testing is simple

Bash

Python

Node.js

Scala

Why is this important?

So what are test frameworks for?

Conclusion

Certifications I'm considering, as an experienced software engineer

Requirements

Certifications I found

Business Analysis

Software testing

Architecture

Other

What I'll choose

2017 freelance development platforms

Codemill

BountySource

Upwork

Toptal

LinkedIn ProFinder

Usage