DEV Community: Kevin Heis

Why I’m Building Sagefy

Kevin Heis — Wed, 21 Aug 2019 22:55:56 +0000

The challenge

The greatest opportunity for technology is human learning. By elevating ourselves, we can accelerate our progress and growth. I’m hoping Sagefy shows a different way we can think about learning with technology.

The most important challenge we can address for learning is ensuring we adapt the content to the learner’s current state of knowledge and skills, as well as their learning goals. Learners waste time and effort when the content doesn’t tightly fit their prior knowledge. If the content references things the learner doesn’t know yet, or if the content repeats information the learner already knows well… that wastes time and effort and lowers motivation. And if the content is outside of the learning goal, the learner loses focus.

Also, we want to make sure that the learner isn’t restricted by topic. Many of our current educational tools make knowledge pay-to-play. We also rely heavily on experts to create selected (limited) content for learners. And where there are open systems with more varied and deep content, the learner doesn’t have a clear path to follow.

We have every reason to want every single person to pursue knowledge in the most time-and-effort efficient means possible. And we want to encourage each person to pursue their interests as deeply as possible.

What Sagefy is

Sagefy is an open-content adaptive learning system.

Open-content means anyone can view, share, create, and edit content. Wikipedia is probably the most well-known example. Open-content means that Sagefy can grow quickly and reach learning subjects not regularly reached by current educational technology.

Like most open-content systems, Sagefy works based on a consensus-based decision making process. Consensus-based decision making means that instead of relying on a single expert to create content or approve changes, a large community of learners can create the content themselves. Consensus-based systems tend to produce content as high of quality as experts, but in a much larger scale.

An adaptive learning system tries to optimize the learning experience based on what the learner already knows and what the learner’s goal is. So instead of a planned and structured course of content, Sagefy presents small pieces of content based on the learner’s responses. By basing the experience on prior knowledge and goals, Sagefy hopes to get the most out of the time and effort spent learning any topic of the learner’s choosing.

We can ensure the content matches the learner’s knowledge and goals by…

Assessing the learner’s current knowledge.
Skipping content the learner already knows well.
Backing up when the learner doesn’t know a prerequisite.
Repeating content when the learner needs review.
Staying focused on content specific to the learner’s goals.

For the time being, I am targeting Sagefy at independent adult learners. This target audience makes Sagefy simpler by not dealing with practical concerns of institutional integration. Younger learners tend to be in a classroom setting, which comes with expectations and challenges that would make Sagefy more difficult to build. Some of these challenges include participation requirements, preset curriculum requirements, testing and grades, mitigating cheating, and instructor analytics. Adult learners in institutional settings faces similar practical challenges as well. By focusing on intrinsically motivated learners, I skip, at least initially, many of the more practical concerns of the more traditional classroom and training scenarios.

Sagefy is completely open-source and freely available. My motivation is to share knowledge more broadly, more effectively, and with more focus. I’m not interested in making a profit. Money is simply a resource. By removing the concerns of profit, I can focus on making Sagefy the best platform I can imagine for learning.

The combination of open-content, adaptive, and free positions Sagefy in a way no other learning technology is. This combination means Sagefy can reach a wide variety of learners, all while attempting to provide an in-depth and effective learning experience.

How Sagefy works

The types of things in Sagefy. There are three types of entities in Sagefy: cards, units, and subjects.

Card. A card is a single learning activity. Examples include a short video, a multiple choice question, or a single math problem. An example question would be, “What is the median of 5, 7, and 10?” Cards are the smallest entity in the Sagefy data structure. A card always belongs to a single unit.

Unit. A unit is a single learning goal. A unit is similar to a short lesson. An example would be, “describe the differences between mean, median, and mode.” A unit is the medium size entity in the Sagefy data structure. A single unit has many cards. A unit can require other units before it. A unit belongs to many subjects.

Subject. A subject is a collection of units and other subjects. Subjects are like classes or courses, anything from like a short workshop to an entire degree program. Examples could be, “measures of central tendency”, “descriptive statistics”, or “first statistics course”. A subject is the large size in the Sagefy system. A subject contains many units and other subjects.

The learner’s experience. For the learner, Sagefy is a simple, focused flow that allows for some choice in the process.

After signing up, the learner finds a subject the learner would like to learn.
Then, once starting the subject, the learner will see the tree of units that represent their progress to the learning goal.
Next, the learner can choose a unit, with one choice recommended automatically.
The learner engages with the content, in the form of videos, audio, written pages, multiple choice questions, short answers… until the learner has learned the unit.
Finally, the learner continues to choose units and learn them until the overall learning goal is complete.

The content creation process. Most open-content sites advertise “anyone can edit.” And it’s a true statement. However, that doesn’t mean you should expect to go to Wikipedia, make changes on a popular page, and still see those changes still there even one minute later. There’s an implied process instead:

Go to the talk page, and describe the changes you’d like to make.
Build a consensus with those who work on that content to make the change.
Resolve any dissenting opinions.
Make your content change.

Discussion is an inherit part of any consensus-driven process. So for Sagefy, rather than allowing direct editing, the content process requires discussion. And where else to post other than in the topic itself?

So to create a new unit, let’s say “describe the differences between mean, median, and mode”, the process in Sagefy would look like:

Go to the discussion section of the relevant subject, and make a type of post called a proposal.
Describe the content you’d like to add by filling out the form. (So there’s no overhead.)
Summon enough people to agree with your proposal. They can do so using the vote post type.
If someone has voted down your proposal, you can work with them to resolve differences or try to create a different proposal instead.
When there are enough approving votes and almost no down votes left, the change will automatically occur. (I have documented the specific math.)
Sagefy keeps a history of all changes made and proposed to every card, unit, and subject.

Because Sagefy is an adaptive learning system, content creators get statistical indicators of how different content performs with real learners. Hopefully, these statistics will help us to improve content and experiment to find teaching opportunities.

The adaptive learning process. We know that one-on-one learning tends to have the highest effectiveness. Why? Because there’s an immediate feedback from the learner to the mentor. Does the learner already know the subject? Skip ahead. Missing something? Let’s back up. Bored? Let’s switch to something else. This interaction is easy to do one-on-one, but basically impossible when there’s multiple learners per mentor. Technology, however, makes this interaction widely scalable.

If you’re interested in getting into specific of what I’ve built so far and planned, I have that documented. I’ve tried to come up with something simple, easy to calculate, and fast on a server, while also general enough to work for a wide variety of topics. I hope someone smarter than me comes along and finds a much better way to do it than I have, but what I have is enough to get started with showing the concept.

Sagefy’s ideas

After spending a few years reading of reading books and articles on educational research, I’ve come to the conclusion there are seven main ideas that come up repeatedly:

Do one thing at a time.
Set and adapt to goals.
Adapt to the learner’s prior knowledge.
Build the graph.
Dive deep, going beyond rote memorization.
Connect the learning experience with real-life examples.
Get learners to learn together.

Do one thing at a time. Multitasking is a well established myth. When we think we’re multitasking, instead what we are really doing is rapidly context-switching. Multitasking increases the time to complete individual tasks. We also can’t really learn more than one concept at a time, either. It’s better to focus on learning one concept, or one integration of concepts, at a single time. “Do one thing” also means cutting clutter and tucking away everything that isn’t the task at hand, creating focus.

Set and adhere to goals. Focus comes up again. When we present content to the learner outside of their goal, the learner loses motivation. Sometimes we do this mistakenly, other times we do this out of passion for the subject. Either way, by clearly articulating the goal the learner, the learner knows what parts of the content are primary, and which parts are supplementary.

Adapt to the learner’s prior knowledge. This idea is the one Sagefy focuses on the most. I’ve iterated this idea quite a bit throughout this article so far. If we task a learner with material they don’t have the prerequisite knowledge for, there’s no place to fit the new information in to. If the learner already knows the material, the learner gets bored and loses motivation. By optimizing for prior knowledge, we can reduce the time and effort it takes to learn new things. Here’s some research resources for further investigation.

Build the graph. Our learning system works not by just absorbing new material, but by connecting existing knowledge to new knowledge. Because of this, its critical to show how new information fits into the learner’s existing schema of what they are learning. Without this organizational map, learners can have difficulty storing the new information effectively.

Dive deep, going beyond rote memorization. It would be silly to expect that someone who just memorized the names of the different joints in the body to then immediately perform knee surgery. Knowing the information isn’t enough. We also need to know how the information relates to other information, as well as how to apply that information to a variety of contexts. Those are separate pieces of knowledge. And its also not enough to expect learners to have something memorized for life after the first time they see the information. The learner needs to review information regularly to keep the memory strong.

Connect the learning experience with real-life examples. How many of us have been in a math class where the teacher goes very deep in abstractions, proofs, etc. without explaining how the concept is relevant? That’s the counter of this principle. When we show the learner the relevance of the material and also make connections to something tangible they already know, the learner integrates the material into their memory more quickly.

Get learners to learn together. By learning together, we can avoid gaps in knowledge. Also, we can work with someone at a similar level of prior knowledge to form knowledge, especially about how we can apply that knowledge to different contexts and different understandings. Of the ideas here, this idea has the most nuance. A learner with little prior knowledge would normally benefit from individual study before joining a group of learners. I haven’t figured out a good way to integrate social learning into Sagefy other than working together to create content. But, social learning is a topic I’d like to pursue more in the future.

There’s not widespread dissent on any of these points, though there are some nuances. Despite the repetition of these points throughout the literature, many of our current learning systems fail to take full advantage of these ideas.

What makes Sagefy different

In this section, I’ll talk about some of the existing learning systems and how Sagefy would compare to them.

Classroom learning. When we think of education, the classroom comes to mind first. The classroom can be a wonderful place to learn, to interact with other learners, and engage the content fully. The best classes motivate and inspire. Unfortunately, many classes don’t meet this bar. With so many learners, it is very difficult to create a system where the the system adapts the content tightly to the learner’s knowledge and goals. The classroom isn’t a free thing by any means, either, limiting its reach. Some teachers do a wonderful job of staying focused, showing examples, showing organization, getting learners to learn together, and going deep with the materials. But not all do. Probably not most.

Learning management systems. Learning management systems, or LMSs, are the most common educational technology used. Schools probably use Blackboard most frequently. The intention of the technology is to support or replace the classroom experience. The LMS’s features are almost completely driven around the classroom model. So most of the same issues with the classroom would apply to LMS, including both scale and adaptability. Most LMSs are not adaptive. SmartSparrow is one example of an LMS that is.

Current adaptive learning systems. Current adaptive learning systems pair adaptive learning algorithms with expert created content. Usually, these systems support either classroom education or training. The adaptive ability, paired with the ability to create highly customized content for the given subject, means these systems currently provide some of the best in class experiences for learners. That said, making an adaptive learning system takes time and effort. I’m not aware of any of these systems that support open-content or are free for public use. I’m also not aware of any that deal with the groups of learners scenario. Some examples of this are Cognitive Tutor, Aleks, LearnSmart, and Knewton.

Massive open online courses. Massively open online courses provide a LMS type of experience for free or low cost to a large audience. A few examples are Khan Academy, Coursera, Duolingo, and EdX. MOOCs scale widely, but still rely on expert created content and generally are not very adaptive. Some have some adaptive or adaptive-like features, like quizzes or practice questions that change in difficulty. Most MOOCs have the same disadvantages as LMSs. The quality varies widely; some of the content is best in class, some of it is thrown together.

There’s a few examples of MOOC-like platforms that do things a little differently. Udacity involves more practice, which is a plus, but still along the same lines. Duolingo focuses on languages, and uses gamification techniques. Khan Academy has a beautiful tree of knowledge, showing prerequisites easily, and again integrating practice heavily into the system. A few of the alternative MOOCs focus on problem-based learning instead of lecture-based.

Flash card systems. Flash cards systems are super common. Usually, these are learner created content, and provided for free. Some of these have some adaptive qualities, like spaced repetition. Spaced repetition means that a learner will review the material after 1 day, then 3 days, then 7 days, and so on, until the material is strongly stored in long-term memory. Flash card system though don’t go beyond memorization. A few examples of these systems are Anki and SuperMemo.

Open content systems. Current open content systems provide deep, high quality content on a variety of topics. However, they do not have adaptive qualities generally. The most common and largest example is Wikipedia.

The backstory of Sagefy

Like most people, I considered the means of learning while attending school. As a technologist, I often considered how I would do things differently, or how to optimize the experience. For every wonderful, inspiring, enriching, and useful course I’ve taken over my life, there have been at least three or four that weren’t. During my time in graduate school, I also had the opportunity of getting to work on educational websites for the University of Oregon.

After finishing school, it became clear immediately to me how many of my peers were struggling. All of us have been through 17 or more years of the education system. 17 years of largely forgotten material. 17 years of preparing to be adults in this society. Many of my peers still, about seven or eight years later, work either as baristas at Starbucks or clerks at the Apple store. I’m sorry, but no one is buying a house on $40k a year when there’s also $30k in student loans. I know, first world problems. I’ve been lucky to find myself in a career track in software, especially given that my degrees are in art and music.

It takes about 17 years in the education system to become considered a contributing adult in our society.

17 years is just too long. 17 years is too long to become a contributing adult. Our expectations of how learning should work and what really works are so far out of sync. (I’m not saying Sagefy is the solution to that, just that we need to be having the conversation.)

There was a time in my life, about 2012 or so, I got hooked on watching TED videos. Yes, yes, a great deal of it is the same inspirational fluff on repeat. Some of it is unique, however. During that time, my life had put me in this place and position. I was just following the course in front of me. And I started to ask myself, what do I really want to do? What’s the best thing I can contribute?

There’s so many huge challenges in front of us, more than any other time in human history. Global warming, energy, water, technological change, population growth, healthcare, debt. Just a few examples. I will never be a environmental scientist, a politician, a mathematician, a neuroscientist, or a rocket engineer. I’m not equipped to meet any specific challenge. But the greatest investment we can make is on learning. If I can’t solve it, I can help someone who can.

I asked myself what the best thing I can contribute would be. I came to the answer that the best investment is on learning.

I started reading on learning science. It naturally fit into my knowledge and skills with user experience research. First, I started mostly with some really great books. From there, I started to get into published articles, both academic and journalistic, about learning science. I started to form ideas of the best learning system I could imagine.

My friend Noah came for a visit for a few days in early 2013. I talked about my reading and my ideas. He asked me, “why don’t you just build it?” At first it didn’t seem possible. But I started thinking about it. I started thinking about the MVP. And even though I’m no expert at any specific part of it, I knew enough to put something together. I started working on Sagefy that April.

I already knew how to build a web API and a web UI. So as I started planning and working on those, I researched the algorithms side. I’ve taken quite a few online courses on different platforms, one of which being Ryan Baker’s Big Data and Education course. Particularly, I understood the Bayesian Knowledge Tracing section and saw a perfect fit for what I was trying to build. Now, an MVP exists and is live, ready for use.

I realized after building the MVP that no one would understand what I built with no content. So the last few months, I’ve been building a small course on Electronic Music, the area of my masters. The first few units are live.

The future of Sagefy

I realize, as much as anyone, that Sagefy most likely won’t succeed. I’m okay with that. There’s so many things that can go wrong, and more importantly, so many right things that may not happen. Lack of interest, difficulty gaining a user base, challenges with creating content, or improving the adaptive algorithms… there’s many challenges ahead. My biggest hope is that, even if Sagefy isn’t successful, that Sagefy may inspire a conversation about learning in this technological era. Sagefy doesn’t have to be big or successful, but some system does need to be successful. The opportunity is too great to ignore. For me, the best thing I can do is get some ideas out there about how to account for how people really learn in a learning system. Even if I fail, I can demonstrate that an alternative is both possible and realistic. Of course, if Sagefy does succeed, even better.

I hope that someday, Sagefy will become a place where anyone can come to learn anything, regardless of what they already know. And then, be able to achieve their learning goal as quickly as possible. Whether its something small, or an entire degree program, Sagefy will have all the information available in a format realistic with how people learn. I realize some subjects we can only teach in person, but there’s so many subjects that we could teach online that we don’t currently.

I foresee a time too where some organizations, like schools, corporations, non-profits, and governments, would want to run their own instances of Sagefy for educational or training purposes. My current financial vision is Sagefy would be donation supported for the public facing instance, and that support services would be available for private installations.

If you’d like to know more about Sagefy, visit today.

Choosing an Open-Source License

Kevin Heis — Wed, 21 Aug 2019 22:54:27 +0000

I have to start this article by saying I’m not a lawyer, and nothing in this article is legal advice. If you are looking for a legal recommendation, ask an attorney.

Why Sagefy is open source

When I started Sagefy back in April of 2013, I wanted to build a system where anyone can learn anything. Where learning is accessible to every learner. Where the experience is completely adapted to the learner’s goals and prior knowledge. Accelerating human learning is the best investment we can make.

I knew I wanted to go open source. Open source software has permanence, collaborative environment, and freedom from the profit motive. My passion for accelerating human learning needs the open source perspective. Financial incentives overwhelm education. The open source mindset and my objectives fit perfectly. I’m a true believer in the power of open source software.

The problem with picking a license

In short, the first decision is if you want to develop your project in open source. For Sagefy, I wanted to go in the open source direction.

Software engineers often default to create their personal projects on Github in public. Yet, tech companies almost always default to building proprietary systems. It’s a strange conflict. The reason for the conflict is between the values of entrepreneurship and the values of the open web.

Once you’ve decided to go open source, the next decision is: choosing a license. For Sagefy, this decision was a more difficult process. The Open Source Initiative lists 78 acceptable open source licenses currently. And that’s only for the code! For content, there’s even more options. Each of these licenses look similar on the surface, but the differences are astronomical. Different values, different concerns, different strategies, different implications, different compatibility.

I first chose AGPL3 for Sagefy. I didn’t know much about the different licenses. I knew I was creating a web service, and AGPL3 covered this use case. I popped the license in, and didn’t think much about the issue for a few months.

Most popular licenses

The first consideration in choosing a license is to go with a popular license. With a popular license, people know what the license means. People know what the restrictions are, and know how to interact with the project. With less common licenses, its more work for contributors to understand the rules. An uncommon license may deter contributors and adoption. Furthermore, popular licenses are popular for good reason: the community vetted them.

So which licenses are the most popular? It’s difficult to say because there’s so many different ways to measure. GitHub is about as neutral and as large as we can get. According to their 2015 blog post, the MIT license is by far the front runner with 44% of all licenses. That’s not surprising to anyone in the open source world. The MIT license is permissive and compatible.

The next two most common are GPL2 and Apache 2. Apache 2 is popular with corporations. Corporations see Apache 2 as a more formal version of the compatible MIT license. GPL2 has a history with Linux. Other popular licenses are GPL3 and BSD 3-clause. Unlicense, BSD 2-clause, LGPL3, and AGPL3 round out the top. Each of those last four are about 1–2% of all licenses each.

Reviewing license options, conflicting values, impacts

I further researched about setting up an open-source project. I realized I knew little about the differences between the licenses. So here’s my non-lawyer take on them. Please note I am NOT A LAWYER. And this is not legal advice. If you have questions, you should talk with an intellectual property attorney.

The permissive licenses: MIT, Apache 2, BSD 2-clause, BSD 3-clause, and Unlicense

If I want a super permissive license or have no idea what I want, I pick MIT. If I am concerned about patents, contributor agreements, or trademarks, I choose Apache 2. Be aware if you pick Apache 2, other open source projects under GPL2 may be reluctant to use your project.

The MIT license is short. The MIT license appeals to the ideal of minimalism. The MIT license pretty much just says:

You can do almost anything you want with this code.
You have to include the license if you redistribute this code.
The author waives any warranties. You’re on your own.

The MIT license is by far the most common for a reason: it’s simple and easy to understand.

Apache 2 is often used for large projects and corporate backed projects. Despite being many times the length of the MIT license, the Apache license is similar. The length includes definitions of legal terms to avoid ambiguities. Apache 2 adds the following to the scope of the MIT license:

The authors are allowing you to use patents related to the code.
If you redistribute the code, you need to state what changes you’ve made.
Contributor works in the project are under the same license.
You can’t use the authors’ trademarks as your own.

Unfortunately patents are a major problem in the software world currently. That’s the main draw of the Apache 2 license. Protecting trademarks are within most corporate interests. And it’s not clear if open source projects need Contributor License Agreements. Apache 2 addresses these concerns.

There’s an unfortunate dark side to the Apache 2 license. GPL2 licensed projects may not be able to consume your project. The GPL*3* licenses are all okay with consuming Apache 2 code. There’s disagreement between different groups about if Apache 2 and GPL*2* are compatible. I don’t agree there’s a compatibility issue. The position of the Free Software Foundation is there is a compatibility issue.

The BSD 2-clause is like the MIT license. As a non-lawyer, I can’t tell the difference, if there is any. The BSD 3-clause adds a third clause. If the original authors don’t endorse your work, you can’t pretend the original authors do. Granted, in real life, I don’t think you can’t make that claim regardless. The 2-clause is actually the younger form. I don’t see much reason to use the 3-clause over the 2-clause. I can’t say I would ever use either BSD license over the MIT license.

Unlicense, which despite its name is a license and is like the MIT license as well. To the scope of the MIT license, the Unlicense includes the following:

A public domain release statement.
Nothing says you need to include the original license.

I can’t imagine there’s much of a practical difference between MIT and Unlicense. Again, speaking as not-a-lawyer.

The copyleft licenses: GPL2, GPL3, LGPL3, AGPL3

The split here comes down to values. If you want anyone to use your software — a person, an organization, a business, whatever — then permissive licenses are the way to go. But, if you want derivatives to stay completely in the realm of open source, GPL may be the right way to go for you. If you choose a copyleft license, many companies will be reluctant to use your software. The essence of a copyleft license is derivative works must use the same license.

GPL2 is the license of Linux. The bullet points:

You can do almost anything you want with this code.
You must include the original license in forks.
Copyleft: Derivatives must be under the same license.
No warranties. You’re on your own.

At a glance, just from these items the main difference from MIT is the copyleft part. The full license does get a bit more technical about specific legal situations.

The first author of Linux feels this license is the best for his work. His reason is something like: “there should only be one true open source license and not a variety of them.” Many large projects use GPL2 instead of GPL3 for this reason.

I agree the decision can be difficult to figure out which license is the right one for a given project. But on further examination, different teams value different things. And that’s okay.

The GPL3 reads like the GPL2 license. GPL3 addresses patents, digital rights management, compatibility with other licenses, and license violations. GPL3 code is able to consume Apache 2 projects. If you want alignment with other projects already doing so, use GPL2. Otherwise, I can’t say I find a compelling reason to use GPL2 over GPL3.

The LGPL3 softens some of the copyleft of the plain GPL3. You can “link” to the project without the copyleft coming into play. We can debate what linking means, but you are giving an “out” to the copyleft provision. The AGPL3 tightens those restrictions: using the project “over a network” triggers copyleft.

GitHub created the website http://choosealicense.com/. This website puts the top 3 licenses as the MIT License, Apache 2, and GPL3. That’s a good representation.

This article doesn’t describe all 78 open source licenses. But maybe one of the other licenses is a better fit for your project. That’s up to you.

My Criteria

I would start out a new project with the following criteria:

Do I want copyleft?
— …and do I care about over network use? AGPL3.
— …do I want alignment with other projects? GPL2.
— Else, GPL3.
Else…
—Do I worry about patents, contributor license agreements, or protecting my brand? Am I not concerned about GPL2 projects? Apache 2.
— Else, MIT.

These would be my criteria. I’m not making a recommendation here. If you need a recommendation, then you need to speak with an attorney.

Chosen license: Apache 2.0

I chose to change Sagefy from AGPL3 to Apache 2.0. I was the only contributor up to that point. I had no concern of getting other contributors to agree to the license change.

Do you want copyleft? That decision makes the biggest difference. I want anyone who wants to use Sagefy to be able to do so. I want schools, other organizations, and corporations to set up their own instances. The copyleft provision makes people nervous. I’m amused no one second guesses using Linux or Wordpress — they’re both GPL2. But for other projects, people worry.

Getting more people to use your project means you’ll get contributions more quickly. You won’t get the same percentage out in the public. 100% for copyleft, less than 100% for permissive. But do you want 100% of a smaller number, or less than 100% of a larger number? Different projects answer differently.

That decision would have led me to MIT by default. For Sagefy, there’s benefit to gaining the extra provisions from Apache 2.0. I have no real concern about GPL2 licensed software consuming Sagefy. Not that I would have a problem with a GPL2 project using Sagefy, just I don’t see that scenario as a likely use case.

The future

Sagefy will most likely stay with Apache 2. If there were to make any change at this point, the most likely would be to the MIT license. I don’t know of any lawsuits with the MIT license specific to patents, contributor license agreements, or trademarks. Perhaps Apache 2 isn’t necessary. Yet, as the questions remain open, that’s the environment we’re in.

The law constantly changes. Both in the body of the law and in the interpretation of the law. Because the law changes, so too will open source licenses. I wish we could have the ultimate, one single true open source license. The MIT license is the closest thing we have currently. But the law will change, and new licenses will adapt to those requirements.

When you’re first making an open source project, choosing a license can be overwhelming. Decide if copyleft is the right decision for your project. If you do want copyleft, you need to decide how strong you want the copyleft. If you don’t want copyleft, you can choose between: a simple permissive license, or an aware permissive license.

Any license you choose, open source software is changing the world faster than anything else. Come on in!

If you’d like to know more about Sagefy, visit today.

How to build an adaptive learning system

Kevin Heis — Tue, 20 Aug 2019 00:00:00 +0000

Have you ever started a course, but thought it was too slow? Or too difficult? Wish you could make it go faster? Felt like you didn't get enough practice to master the content? Adaptive learning systems seek to address these challenges.

In this article, I’ll go over what adaptive learning systems are. I cover some background on why adaptive learning systems have the structure they do. I’ll introduce a few adaptive learning systems. Then, I talk about the four elements. And how you can architect an adaptive learning system. We’ll wrap up with evaluating the pros and cons of adaptive learning.

What is an adaptive learning system?

An adaptive learning system is software where algorithms optimize the content to adjust for the learner’s goals and current state of knowledge.

In a traditional e-learning course, you will linearly follow the path an instructor creates. You watch videos, read articles, take quizzes, and practice interactive modules in a predetermined ordered. An adaptive learning system will contain the same types of materials. But the order will change for each learner. The system decides which content to show the learner based on two things:

If the learner’s goal is only a subset of the content, the system can limit the content.
The prior knowledge also comes into play. If the system determines the current path is too easy, the system can speed up to more challenging material. If the system finds out the current path is too difficult, the system may… intervene and review prerequisite content, reduce the challenge, or slow down the pace.

Some related topics include intelligent tutors, adaptive testing, psychometrics, personalized learning, and smart teaching. Many of these topics share algorithms and structures with adaptive learning systems.

Knowledge is a graph: neuroscience

I’m going to start with a little background. This will create context for why adaptive learning systems have the four elements below. The point for this is knowledge is a graph.

The human brain has 86 billion neurons. Every neuron has dendrites, a soma, and an axon.

The dendrites are the input. The edges of the dendrites receive neurotransmitters from the synapse. The synapse is a gap between two neurons.
The soma is the throughput. The soma – which contains the cell nucleus – routes the input from the dendrites.
The axon is the output. The axon transmits an action potential – an electrical signal – to the axon terminals. A myelin sheath covers the axon to protect the signal. The axon terminals release neurotranmitters into the synapse.

Information your brain receives and processes corresponds with a neural pathway. Your brain with myelinate that pathway – strengthen the myelin around the axon to support electrical signals. Because of the strengthened myelin, this path will be more likely to fire in the future. In other words, you learn.

Even in the smallest scale, our brain is a massive graph of connected neurons. We learn and optimize by making some paths more likely to connect than other paths.

Knowledge is a graph: learning science

The strongest predictor of how we perform in a learning environment is our prior knowledge. What we already know before we start the learning experience. A notable psychology paper – 1999 Dochy, Segers, and Buehl – found prior knowledge is 81% of outcome differences between learners. Reviewing prior knowledge before showing new information impacts learning outcomes. And connecting new knowledge to prior knowledge while teaching can have a big impact too. (See Eight Ideas for sources.)

The most famous psychology paper is 1956 “The Magical Number Seven, Plus or Minus Two” by George Miller. The paper suggests that humans have a limited working memory. Miller found for simple numbers, a human could work with about seven items at once. Later researchers found for more complex information, that limit is closer to four.

Some psychologists suggest of these “four slots”, for us to learn, at least one or two must be prior knowledge. How much prior knowledge we can “load up” into one of the four slots depends on the strength of the connections in the graph. When we have both prior knowledge and new knowledge in our working memory, we associate the information. And we strengthen the connection between the two. Trying to learn new information without connecting to prior knowledge limits the strength of the memory.

In short, we learn by connecting prior knowledge to new information. And those connections form a large, endless graph of knowledge.

A few important adaptive learning systems

This section is more context, but optional. I’m not writing an thorough article about the history of these systems, but here’s some bullets:

One of the earliest implementations was the Skinner teaching machine.
During the 1960s and 1970s, there were several attempts at computerized instructional systems. Costs and slower machines limited the success of these systems.
During the late 70s and early 80s, Item Response Theory enabled test makers to start work on computerized adaptive testing.
An early and influential computerized system was the Lisp tutor, also known as LISPITS (1983) at Carnegie Mellon University.
SuperMemo, released in 1985, incorporated spaced learning into a computerized system.
Also in 1985 came paper for Knowledge Spaces, which forms the foundations of one of the four elements.
ALEKS Math tutor came out in 1994, heavily promoting its use of knowledge spaces.
In 1995, Corbett and Anderson published “Knowledge tracing”, forming the foundation for Bayesian knowledge tracing models.
Some important software includes AutoTutor, ACT-R, and Cognitive Tutor Authoring Tools.
Knewton is an example of contemporary adaptive learning systems. Kaplan and Pearson both use Knewton to provide adaptive learning experiences.

The four elements

Most adaptive learning systems today have these four elements. The terms change and so do their scope. But you will almost always find all four elements.

These elements are:

The expert – a graphical model of the “ideal” state, of everything the person could learn using this system.
The learner – a model of the learner’s current state, which shows how likely the learner is to know each of the nodes in the expert graph.
The tutor – the algorithms that determines what content to show and when. The expert model and the learner model inform the tutor. The tutor seeks to optimize content for relevance, challenge, and efficiency.
The interface – which is how to display the learning experience to the learner. In many adaptive learning experience, the interface changes based on the learner model and the tutor’s goals.

Let’s go into each element.

The expert — the big graph of everything

The expert model is a large, connected graph of everything you want the learners to know. As the name suggests, you have an expert on the topic – or experts on topics – create the model. This model is static. The expert model only changes when the scope of learning outcomes change. Or when problems and opportunities to refine the adaptive learning system arises. Most of the work of the expert model is at the beginning of building a new learning experience. The adaptive learning system will access the expert model to compare the learner’s current state with the expert model. The system will also access the expert model to determine which learning experience to focus on next.

Usually, a team of experts will define the scope of learning outcomes. Each node in the expert model should have the following attributes:

A name
A short description, which indicates which skills are under test and what is outside the scope
A list of prerequisite nodes – these form the “edges” of the graph. These prerequisites cannot form a “cycle” – a loop of nodes.

Expert models perform better when each node is small and narrowly defined. For example, each skill in Bloom’s taxonomy – recognition, understanding, application, analysis, synthesis, and evaluation – could each be its own node in the expert model. The combination of two underlying skills should also be a separate node.

There is an endless number of formats you could use to create an expert model, such as XML, JSON, CSV, or YAML. It can help to be able to display the expert model graphically for review.

Some systems will automatically generate an expert model by querying experts in a series of questions, like a wizard. Others will cluster existing learning content, using algorithms like k-means clustering. You may want to review the wikipedia article on Knowledge spaces for a more mathematical description.

The learner — where you are versus where you want to be

The learner element is a model of the learner’s current state of ability. So for each given node in the expert graph, the learner model has a probability associated with it: 1-99%. The system updates this graph every time the learner performs an activity. If a learner answers a question correctly, the probability increases. If the learner answers incorrectly, the probability decreases. Each learner has their own learner model. So each time there’s a new learner in the system, there’s a new learner model. Later, the tutor will use the learner model to decide how to order the learning content.

There’s many algorithms for updating the learner model. Knowledge spaces suggest that as a learner develops a skill, the probabilities for related skills should also adjust. Some adaptive learning systems use simple heuristic models for updating skill probabilities. In item response theory, the probability updates along a sigmoid curve. In Bayesian knowledge tracing, this curve has a more conservative shape. Each model tends to account for these factors:

Before the learner does anything, what do we estimate the probability to be?
How likely is a learner to guess the right answer if they don’t know the skill?
How likely is a learner to slip up even if they know the skill?
How likely is the learner to have “learned” the skill after seeing the item?
How likely does this activity categorize the learner as skilled or unskilled?
How difficult will this item be for this particular learner?

For both item response theory and Bayesian knowledge tracing, you’ll need a means to estimate these parameters. This is one of the most rapidly developing areas in adaptive learning systems, so I can’t make any specific recommendations yet. There’s also researchers creating models with classic machine learning, such as neural networks.

The tutor — what to show when

The tutor chooses which order to select the activities the learner will engage. After each update to the learner model, the tutor will update the path it will take to optimize for that learner. The goal of the tutor is to get the learner to a complete expert graph in the smallest amount of time. Some systems allow learners to focus only on some areas while ignoring the rest. As the learner model is unique per learner, so too is the paths the tutor will take. While the expert and learner elements are data with some algorithms, the tutor is algorithms with some data.

The tutor may decide both which skills to focus on and which activities to have the learner perform. For the skills to focus on, the tutor will often choose skills with the largest impact on the larger graph. This often means focusing on more elementary skills before more advanced skills. For activities:

The tutor will try to choose the most relevant activities to the learner
The tutor will choose activities that are challenging, but not too difficult for the learner.
The tutor will try to choose activities in a way that reduce the total time towards mastery.

Simple adaptive learning tutors may choose activities within a skill at random. Item response theory based tutors emphasize choosing activities that are challenging. In Bayesian knowledge tracing models the market has many different tutor algorithms. Researchers have focused more on the expert and learner elements. So we don’t know what produces the best learning outcomes for the tutor element.

The interface — how to show it

Some adaptive learning systems will change the user interface. As the learner is less familiar with a skill, the interface would reduce and focus more on the task at hand. As learner ability grows, more of the full interface comes together. Some call this process “scaffolding”.

In some systems, learners may ask for and receive hints. When to offer hints and the depth of those hints can adjust based on learner ability.

There’s also some other questions like:

Do you display the expert graph to the learner?
Do you display their progress on all skills? How?
Do you display their progress on specific skills? How?
Does the learner get choices in learning content? Or does the system decide everything?

Depending on the needs of the system, some of these items may impact learning outcomes.

How do we know if adaptive learning is any good?

As these systems come from academia, we have a significant amount of data and history with each system.

Human individual tutoring has the strongest learning outcomes. This is a common finding in educational research. So far, no computerized adaptive learning system has outperformed human one-on-one tutoring.

Researchers have investigated classroom learning alone, computerized adaptive learning alone, as well as combined classroom and adaptive learning. A 2016 paper “Effectiveness of Intelligent Tutoring Systems” provides a meta analysis of these studies. Adaptive learning systems usually outperform traditional classroom learning. Combined with classroom learning, adaptive learning systems create a positive effect, but there are some limitations.

Adaptive systems do particularly well with instant feedback and ensuring skill mastery. Investigators note some areas for improvement:

The cost of developing content for these systems is high.
These systems often can’t contextualize learning the way a human can.
Adaptive learning systems can feel more challenging, which can reduce learner motivation.

Wrap up

Welp, I’ve nerded out now. I’ve covered what adaptive learning systems are. I’ve provided some context for the design of these systems. A touch of history. I’ve covered the four major elements: the expert, the learner, the tutor, and the interface. Hopefully it wasn’t too technical.

Obligatory end-of-article call-to-action: Check out Sagefy, the open-content adaptive learning system I’m working on.

A New Sagefy

Kevin Heis — Wed, 31 Jul 2019 00:00:00 +0000

In this post, I’m going to talk about what Sagefy is. Where Sagefy was in October last year. And why and how I changed Sagefy in the time since. I wanted to share how this big change went down. And document the changes for my future self.

What is Sagefy

Sagefy is an open-content adaptive learning platform. Anyone can learn anything, and Sagefy adapts the content to the learner’s knowledge and goals. Sagefy is open-content, meaning anyone can view, create, edit, and share learning content. Sagefy is adaptive, meaning as you learn, Sagefy changes which learning activities it picks to fit your needs. Sagefy is open-source and freely available.

Where Sagefy was in October 2018

I’ve been working on Sagefy since April 2013.

A 2017 image of the homepage.	A 2018 image of the choose unit page.

Sagefy in October 2018 had lots of features. It was quite capable. Despite being operational for a few years, there was litle human traffic to the site each day. There was about one account sign up per day. And in this version, as users needed accounts for most parts of the site, sign ups were a decent indicator of use. Which is to say, not much use at all.

I’m a fan of usability testing and user interviews. Sagefy has had many of each. The feedback I got was:

I don’t want to make an account to try it out.
It doesn’t have the learning content that I want.
I don’t understand what units and subjects are. (At the time there were 3 types of things: cards, units, and subjects.)
I don’t understand how to make content. (And this was after several iterations of the content creation parts of the site.)

Additionally, referral traffic was very low. Some of the technical choices made it difficult to optimize for search engines.

On the technical side, I had an isomorphic rendering – JavaScript on client and server – with a small virtual-dom library. I had a Python server running REST endpoints with lots of raw SQL and home-built code. And endpoint logic to manage and update in several places with each change. Lots of endpoint logic. And I had three databases, PostgresQL, Redis, and Elasticsearch. All sitting behind an Nginx instance. Test builds were slow, but thorough. In later iterations I was using Docker and Docker Compose. This was all gravy, don’t get me wrong. It worked, and performed reasonably well. But change wasn’t always easy.

Why I wanted to change Sagefy

I decided in October 2018 it was time for a bigger change. I’m the first to argue for aggressive refactoring instead of rewriting a whole application. But as Sagefy is a side project, and with low traffic. The risks were lower with a rewrite. I also spent quite a bit of time the prior months trying to iterate the architecture to support anonymous – not logged in – users learning and making content. It was a ton of work, with no obvious end in sight.

So with “fuck it, I’m going to rewrite Sagefy” being where I was, here were my goals:

Make Sagefy work with or without an account.
Make it easier to make new content.
Find a way to get lots of content quickly.
Focus on leading people to the site with content that will get links and search engine traffic.
Reducing the number of concepts and features.
Figure out how to write way less code to make changes faster, so I can adapt to user feedback faster.

How the Sagefy changed

Mocks

My first action was to create some mocks for Sagefy. I wanted to figure out where I was going. I wanted to iterate fast.

I did these in plain HTML, with the world’s most basic CSS sheet. I know HTML better than any design tool at this point. I used no CSS classes, only the standard tags with light inline styling. I published them on GitHub Pages for hosting. And I wanted to get feedback early, before any real technical investment.

I started with the home page and subject pages, as those are the two pages that will start the user experience. I focused on flows rather than features. Ensuring the user always has a clear next action to take. I also wanted to make sure when people end up in “null states”, there’s something for them to do. For example, today when you search for a subject in Sagefy and you can’t find anything: it prompts you to go ahead and make a new subject. While keeping the above goals and mind. And editing. Lots of editing out features and capabilities. I focused on the main goal: to let anyone learn anything they want, adapted and free.

I requested review and feedback from my email list, and I had 3 users take the time to investigate, review, and provide feedback. (Thank you!) Every point was valid and helped focus the mocks even further. I had some former colleagues and friends review the mocks as well. With the mocks being basic HTML and CSS, I could iterate with each round of feedback fast.

Database and GraphQL service

The thought of having to rebuild everything the way it was in V1 was overwhelming. So much code! The thought occurred to me: I only want to define a schema that describe the whole application. And then from the schema come to life. That’s about the least amount of code you can write and make something reasonably useful.

I looked at some tools on GitHub that were trying to be exactly that… but the finished products were not amazing. Many of these tools were using advanced PostgresQL features to define their schemas. Then I started seeing tools that made entire APIs from the PostgresQL schema: Hasura, PostgREST, and where I am now, which is Postgraphile. When I read the schema design doc on Postgraphile’s website, I felt immediately this would work. The business logic is entirely encoded in Postgres – including authentication and access control. PostgresQL is in practice the only major database that could pull something like this off. Postgraphile makes a GraphQL endpoint automatically from this. I found this appealing. I like both REST and GraphQL. But GraphQL would mean iterating later would often mean no schema change and no new service code for new features. It was ideal.

Another advantage would be cutting down from three databases to one. Postgres and JWTs would handle everything Redis did to manage authentication and caching. Postgres can also do full text searching at nearly the speed of Elasticsearch.

I foolishly decided to try to write the entire database schema up front. This was going on while the mocks were iterating too. My first pass was a draft. I tried to think through the business logic. And I tried to figure out how it would work in PostgresQL and Postgraphile. My second pass was simply trying to make it valid SQL code that the database wouldn’t error trying to load up. I got through that.

Then I tried loading up Postgraphile for the first time against the database. It actually worked. But the schema it generated didn’t make sense to me in a lot of ways. I made some assumptions about how Postgraphile worked, that didn’t turn out to be true. Postgraphile actually does quite a bit more than I thought it would. It predicted lots of little things too, like wanting names to in underscores in PGSQL but camelcase in GraphQL.

February 2019

I realized finishing the whole site and then launching it was never going to happen. It would be too much work. I should have known better, but I was eager. With so little usage, I decided to hard pull the plug on V1 and start over. I redeployed the site with only a home page. Some static text about what Sagefy is, a few links, and an email newsletter sign up.

Serving HTML

I focused on front-end code for a long time. I hate fighting the browser. I love the browser. I wanted to go along with how the browser wants to do things instead of trying to force to go a certain way. This would mean less code and faster iteration.

For the first V2 deployment, I only had the client side running. I decided on JavaScript again for the HTML server. Postgraphile itself is in JavaScript, and its the language I know best. Sure, its a weird, messy language. But its the most popular programming language. So lots of people know it. And the packages available are amazing. And performance-wise, its nearly in the range of Go or Java. Without having to deal with lower level issues, like pointers or verbose static typing.

For my templating language… I chose React. I figured to get started, I would do a simple, server-sends-all-the-HTML sort of site. There’s no front end, browser facing JavaScript at all. I know React super well, and there’s tons of other people that do too. Its tooling is excellent, and so is the ESLint plugin for it. There’s low risk of injection or having an extra close tag. And performance wise, React on the server is about as fast as any other JavaScript HTML templating library. I figured eventually if I wanted to do isomorphic again, it would be easy to convert. That hasn’t happened though. I’ve been able to build everything with plain old HTML and no JavaScript running in the browser.

I did the mocks with the idea that I would make a more designed looking site based on them. But instead, I kept the bare-minimum CSS sheet I had. I converted it to its own repository so I could reuse it throughout Sagefy’s docs and my personal site too. I also took out my inline styles and made them into a handful of tiny utilities, like centering text or inlining lists. This approach makes change super easy. I use basic HTML tags. I don’t have to remember a bunch of classes to make the site look decent. Lots of the popular style libraries avoid changes on basic HTML tags to try to be neutral. By having an opinion, I can dramatically reduce how much CSS there is to manage. The site looks consistent without me trying much at all. I can change things so easily, because the site is so basic. And the performance is of course amazing. No custom fonts to load either.

Feature-by-feature

After that initial page was up, I rebuilt the site feature by feature. The whole technical stack was working.

The first feature I did was restoring accounts, so that people could sign up, log in, and log out again. This was the first feature because everything else later would depend on the authentication model.

Each feature had a pattern.

First, I write the database migration. I’m using a simple Go tool to manage database migrations. The tool operates on straight up raw SQL, no other languages required to use it. The database migrations were easiest to do first in the PostgresQL command line REPL. Then once I get them working I formalize them into a migration.
After that, I’d use the GraphiQL instance that comes with Postgraphile to write the GraphQL query. This was to ensure the database did what I thought it would.
Once the query was working, it was on to the client. I start with the Express endpoint, making the appropriate calls to to GraphQL server and handling the result.
Then I’d write the page in basic HTML with React.

As of this writing, I’ve avoiding DRYing the client code up too quickly. So there’s still lots of copy-paste code. I’ve learned over the years to wait a little on getting DRY. If you abstract too early, then the abstraction needed won’t be as obvious.

This pattern worked for every feature afterwards. And its very little code for the amount of capability it has. Most of the working code is either in PostgresQL … which SQL can do an amazing amount with little code … or in Express endpoints, which I want to work to reduce that even more and rely on PostgresQL instead.

Then the feature order was…

After the basic account flows existed, I prioritized getting subjects on the site. I used some samples I paid people to make on mechanical turk a few years ago.
Searching for subjects.
Being able to add those subjects to your account.
Create a new subject… whether you have an account or not.
Uploading the cards I had from the previous version.
Creating the learning experience again. I used a simplified Bayesian model with statically set parameters to get something out the door.
Creating cards for subjects, and prompting users to make cards when a subject doesn’t have any or enough.
Optimizing pages for search engine indexing and linking.
Created talk pages for cards and subjects.

(If you want to read more about it…)

Content

At this point, I addressed many of the goals already:

Sagefy works with or without an account.
Its easy to make new content, and Sagefy prompts users to make content when appropriate.
Reduced the number of concepts and features: units were gone, as well as many of the database tables that supported V1 features. Units are now subjects with no child subjects, so now there’s only two types of things in the system.
Way higher search engine optimization and social media linking capabilities.
Way less code to make changes faster, making new features very fast to finish and iterate.

The big one I didn’t have yet was:

Users can’t find the content they want.

So I got a little “creative”. I used the Library of Congress system to create a list of subjects and their parent/child relationships. I used the Library of Congress due to its thoroughness. The Library of Congress system is much more systematic than Dewey decimel or other systems. Wikipedia isn’t quite a structured so there’s both not as many but too many subjects at the same time. Then with that list, I used Amazon Mechanical Turk. Again, maybe not my favorite system but with almost no budget, you make compromises. I had the workers pull some basic information about each topic: a definition, a few subtopics, a few scholars in the area, and why you’d want to learn about the topic. The results are pretty good. And I got those uploaded to the site, which means Sagefy as 1500+ new subjects.

But the subjects were all empty. I didn’t want to rely on Mechanical Turk again. It would have been too expensive to go down that road for cards anyway. So I learned some natural language processing. I made a small Python script to search Wikipedia to find the matching page for the subject, summarize it, make multiple choice questions. And then also called other APIs to get a YouTube and a Vimeo video. The quality is pretty variable, but its better than nothing for sure.

Remaining features

Now that there was some content to get indexed and draw more people to the site, I went back to wrap up the remaining features in the mocks:

Editing subjects and cards
History pages for subjects and cards
Site wide search
Viewing contributions on the dashboard

Where Sagefy is in July 2019

1. Search for a subject	2. Find your subject	3. Choose the next step	4. Learn!

Sagefy has every feature that was in the mocks. It’s got lots of content, covering almost every significant academic subject. And some cards in each subject as well. The quality of the content is highly variable. Sagefy is available to all users – with or without an account. And available to search engines and linkable. Every day the traffic is growing.

I’m satisfied with the change. I’m happy I did a rewrite, even though its usually a bad strategy. I’m happy I focused on user feedback instead of dreaming. I’m glad I started with very basic mocks. And even though its sort of compromising, focusing on search engines and social media linking does produce results. It took some time to figure out how to do this with the least amount of code, but it was time well spent.

If I were to do it again, I would do even more usability testing and feedback rounds. I would not try writing the whole database schema up front. Instead I would start from day one going feature-by-feature, flow-by-flow. Its so easy to change if you make a mistake, especially at small traffic. And finally, I would have created content earlier, to get more traffic earlier on while I was building features. Instead of the reverse. Content is what gets people to your site, after all.

Where Sagefy will go from here

I have no idea :) Now seems like a good time to pull back a little on the project. Observe how it does in the wild for a few weeks. After that, I’m going to go back to usability testing and interviews. And look over traffic data. This will help me to find out where the next biggest opportunities are. Sagefy is far from perfect for sure. The content could be much better, and the code could use some refactoring and automated tests. But I know when those new opportunities come, its going to be much easier and faster to change. And hopefully I won’t need another rewrite for a long time :)

Obligatory end-of-article call-to-action:

Visit Sagefy. Feedback is welcome!
For more information, watch this 3 minute YouTube video or read the docs.
Want to help? Check out our “Want to Help?” doc.
Sign up for updates.
Reach out to Sagefy on Twitter.

How I Write React/Redux Projects in 2018

Kevin Heis — Thu, 30 Aug 2018 00:00:00 +0000

I’ve been in JavaScript development for about thirteen years. I’ve been through Prototype.js, YUI, MooTools, jQuery, Backbone, Angular, and now React (and countless other libraries and tools). I’ve been mostly working on React/Redux browser projects for the last four years on a variety of projects.

This article documents the strategies I use when building browser projects in React/Redux. These practices are the result of: Lots of trial and error. Reaching consensus in various teams. And observing where problems tend to happen. I assume here you’ve at least read the official React docs and Redux docs. If you haven’t yet, do that first.

Nothing in here is a rule; these are high-level guides. Take what’s relevant to you and ignore the rest. Every project has its exceptions. Some projects, React/Redux would be overkill. (I love a plain HTML site.) Or the project requirements are too specific, and React/Redux would get in the way. I don’t love everything about this system. But for a majority of web projects, React/Redux is a decent default.

The “in 2018” part is relevant. Every few months there’s a shiny way of doing things. I’m sure the 2019 version of this article would be quite a bit different.

Please don’t use this guide to try to override a team consensus. I’ve never seen a project fail due to technical choices; but I’ve definitely seen a project fail when the team couldn’t work together.

Why React & Redux

Why use React? There is a sea of options for web browser projects. A few of the key benefits of React:

React is the most popular option now by far. The largest number of new projects use React. React will continue to have support for at least another 5 years or so. Popularity also means its easy to get help, and there’s a large system of tools supporting React.
Vue is growing in popularity. Vue uses a similar model to React. Maybe in a few years Vue will become React’s successor, but my guess is React’s successor doesn’t exist yet.
One-way data flow is a vast improvement over previous systems. The closer to follow this model, the easier the code is to work with.
The virtual DOM is a performance benefit. It’s an even larger maintenance benefit.

There are many other virtual DOMs… some are smaller in filesize, some are faster. But React is mature, and those other gains are small in comparison to what you get with React’s maturity.

Why use Redux? There’s many options here too for state management.

Redux is small and easy to understand.
Redux is the most popular system for state management right now.
Redux helps keep your changes to state pure , making changes easier to follow and understand.
Flux, React’s original state management system, gets complicated quickly.

Many other options exist for state managment, MobX likely the most popular. Redux has a problem with boilerplate. Most of the other options have specific ways of addressing this boilerplate. The other options might make sense for your project. My bet is that there will never be a true consensus on the best way to reduce Redux’s boilerplate. Redux is a general solution. And about as specific as we can get while making sure it will work in a variety of situations.

Tiny projects are better off only using React’s component state on the highest component in the tree. You can switch to Redux or another system later.

There’s some other libraries I recommend using. You need react-dom for web projects if you are using React. react-redux is a tiny library that gives a little more guidance on how to integrate React and Redux together. For projects that have more than one route, react-router with react-router-dom is the most popular option. They aren’t perfect, and have changed a great deal in a short time. That said, they’ve come to a good pattern for routing.

Overarching Principles

There’s a few principles I’ve developed over the years working on these projects. These principles can help avoid problems. That said, there’s never an ‘always-right answer.’

Do the boring way first. There’s so many tools and libraries out there in JavaScript world. My general advice is see if you can meet the requirement with a basic function. Do that until you’ve demonstrated that won’t work. If you can use something already built into the browser, try that first. If its specific to React or Redux, try it the way the authors suggest until you know that doesn’t work for your project. (And if you can build your project without JS… don’t assume you need JS at all!)

Functions > Classes. I’m not a functional programming purist. Sometimes mutable data models the problem perfectly. Classes are great for situations where you will need multiple copies of some type of data, and each instance has its own properties. But unless you know you need mutable data or a class in the situation, pure functions are the better default. Functions are simpler to compose, easier to test, and quicker to read.

YAGNI > DRY . Duplicate code is usually bad. But duplicate code isn’t as bad as having overly abstracted code, or wrongly abstracted code. I’ve taken the approach of letting some duplication slide until the pattern is obvious. Most bad abstractions come from trying to be DRY before the pattern is obvious. A little patience is valuable. And sometimes, if there’s little differences each time, you may be better off leaving it ‘wet.’

Single-source of truth > DRY. Redux gives you a single state to work with. Having your Redux state be the single source of truth is more important than being DRY. There will be times you have to decide between having a single source of truth or obeying DRY. Take the single source of truth approach. As long as you have a single source of truth, the pain of not being fully DRY is bearable.

Base → Everything Else → Index. Index is often used as the ‘root’ file name for a directory. I’ve seen cases where other files in the directory depend on index. And others where index consumes the other files. The latter is less problematic. For shared values and functions, I will store them in a “base” file. My other files consume the base file. Then the index file consumes all the other files.

Be consistent with names. JavaScript defaults to camel-case names. Unless you have a good reason to, stick with camel-case as much as possible. You want the name of things to be as consistent as possible. The component, its styles, its filename, its CSS class name, its associated selectors… should all use the same name. It’s confusing when they don’t line up.

Configuring Local Development

Almost goes without saying to use Git for version control from day one.

I would start with **create-react-app**. Unless you have no company boilerplate or other configuration you need to start with. The authors of CRA work on React as well. CRA is an easy way to get started. And there’s an easy opt-out mechanism if your project needs it later. Having seen many boilerplate systems over the years, CRA is the nicest I’ve seen. next.js might make sense for your project too. If you aren’t using a boilerplate, you’ll need to install and configure Babel and Webpack yourself.

Other than a quick demo project, I would recommend using **eslint**. The static analysis tools that come with eslint rival most statically typed languages. I usually opt for the AirBnB ruleset — full for React projects, base otherwise. And I also add in the prettier exceptions. There’s other configurations out there too that might work better for your project.

{
  "extends": ["airbnb", "prettier", "prettier/react"],
  "env": {
    "node": true,
    "browser": true,
    "jest": true
  },
  "parserOptions": {
    "ecmaVersion": 6,
    "sourceType": "module",
    "ecmaFeatures": {
      "jsx": true
    }
  }
}

Speaking of **prettier** ** …** much better than arguing with teammates about code style. And it reduces the number of lint warnings you have to deal with. I’ve been using semi-colon free style. In practice semi-colon style makes almost no difference either way.

semi: false
singleQuote: true
trailingComma: es5

Even if you aren’t using React/Redux, I would recommend using **lodash.get** . Heavily. Branch coverage in JavaScript can become a real problem. Having to write and test every possible branch will make your tests much more brittle. JavaScript usually operates in an environment where almost anything can happen. There are rarely any guarantees. lodash.get is one of the tools to help deal with that reality.

// before lodash.get
const userEmail = state &&
                  state.users &&
                  state.users[userId] &&
                  state.users[userId].email ||
                  ''

// after lodash.get
const userEmail = get(state, ['users', userId, 'email'], '')

There’s many styling tool options today. In recent years I’ve worked with sass, stylus, and glamor the most. Every project has different requirements here so I can’t make a general recommendation. If you are starting a new project, I would drop in normalize.css and then have a single CSS file. You can change to something else when the file becomes too large. You could also look at a full CSS system like tachyons or Bootstrap if your project doesn’t have too specific design requirements.

Localization isn’t a solved problem. The browser built-in window.Intl can get you pretty far by itself.

Directory Structure

My directory structure will look something like this:

README.md
package.json
client.js
server.js
state/
data/
services/
middleware/
selectors/
views/components/
views/containers/
views/routes/
constants/
helpers/
images/

Root Files

Every project should have a README. Your README serves two purposes: a) documentation for developers and b) marketing. I would suggest your README contains at a least…

The name of the project.
A short description. (Who is using this, what are they doing with it, why are they using this.)
A screenshot, if that’s relevant.
The build status of the master branch.
Links to documentation of the tools and APIs your project consumes.
A list of the directories in the project, and what their intended purpose is.
A list of top level npm/yarn commands and what they do.

Your boilerplate may or may not include a **package.json** for you to use. The JS community used to do Grunt and then later Gulp. But now there seems to be a consensus to use the scripts section of package.json. You’ll have development, build, and test commands. package.json now also serves as your home for tooling configuration. I could write a whole article about only this one file, so for this article that’s all I’m going to say about package.json.

{
  "name": "messages",
  "scripts": {
    "build": "# ...",
    "start": "# ...",
    "test": "# ...",
  },
  "jest": {
    ...
  },
  "dependencies": {
    ...
  },
  "devDependencies": {
    ...
  }
}

If your project is client side only — has no SEO requirements — then a single client.js entry to your application will be fine. Otherwise, you’ll need a **client.js** and a **server.js**.

**client.js** will each be responsible for:

Creating your Redux store,
Integrating your Redux middleware, and
Binding your store to React with Provider from react-redux, and
Binding ReactDOM to your actual DOM.

If you are using React Router, then BrowserRouter will be in this file.

const store = createStore(reducer, applyMiddleware(logger))

document.addEventListener('DOMContentLoaded', () => {
  const container = document.querySelector('#react-root')
  ReactDOM.render(
    <Provider store={store}>
      <Router>
        <Messages />
      </Router>
    </Provider>,
    container
  )
})

(All code samples in this article are for demonstration. They do not have everything needed for a production experience.)

Your **server.js** file will need to

Create a new Redux store on each request, which again means binding middleware to the store.
You’ll need to make any necessary service calls here to generate your Redux state.
If you are using React Router, then StaticRouter will be in this file.
You’ll also need your base HTML markup.
Then you’ll put the finished state and initial React HTML,— rendered with react-dom/server — into your HTML template.

For actually handling requests, I would recommend Express. Express is easy to use and well supported, and perfect for this sort of situation.

const app = express()

const html = `
<!doctype html>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Messages</title>
<link rel="stylesheet" href="/index.css">
<body>
<div id="react-root">{innerHtml}</div>
<script>window.preload={state}</script>
<script src="https://cdn.polyfill.io/v2/polyfill.min.js"></script>
<script src="/index2.js"></script>
</body>
`

app.get(/.*/, (request, response) => {
  const store = createStore(reducer, applyMiddleware(logger))
  // once the store is filled of data, then...
  const myContext = {}
  const innerHtml = ReactDOMServer.renderToString(
    <Provider store={store}>
      <Router location={request.url} context={myContext}>
        <Messages />
      </Router>
    </Provider>
  )
  if (myContext.url) {
    response.redirect(myContext.url)
  } else {
    response.status(200).send(
      html.replace('{innerHtml}', innerHtml)
          .replace('{state}', JSON.stringify(store.getState()))
  }
})

app.listen(3000, () => {
  console.log('serving app realness')
})

Server-side rendering React/Redux is difficult, and more than I can describe in this article. There’s lots that can go wrong with server-side rendering. If you don’t need to do server-side rendering, I wouldn’t. Otherwise, there’s several articles on Medium about how to approach server-side rendering with React. The rule is you need to create everything per request… unless you are caching.

If you have any configuration that you need to have in both the server and client, such as API endpoint URLs or image base paths, I would have that in server.js. Expose it through the HTML document to the browser with an inline script tag. That way, all your configuration stays in one place.

If your needs start to expand on the server-side, I would recommend using the Express middleware pattern. This can help keep your code small and modular on the server-side.

Cross-browser support has been easier the last few years, but there’s still difficulty to know what features each browser is going to support. polyfill.io is an easy way to resolve that issue. You add one script tag to your HTML before your own scripts, and you’re done. Highly recommended.

State

The state is the heart of your React/Redux project. If you master your Redux state, the rest of your application will flow from there. If you get it wrong, you’ll spend lots of time downstream trying to correct the issues.

Think of your Redux state as you would a SQL database. There are three main guides:

No duplicate data.
No derivative data.
Be as flat as possible.

The easiest way I know to abide by these three guides is to write out a state schema. You could have a single file in your state/ directory that describes the keys and default values for your entire state tree. You also add comments to describe what the fields are. And if you wanted to go a little extra, you can also add validation rules to your state schema, though I don’t think that’s usually necessary.

If you have the same fields in two different collections, then you should separate out the shared fields into their own collection. This avoids duplicate data.

Derivative data is data you could figure out from one or more other pieces of data you already have. For example, if you have a firstName field, and a lastName field, you don’t need a fullName field.

When you have nested data, it’s harder to work with later on. As long as you aren’t creating duplicate or derivative data, you want to flatten your schema as much as possible.

The way I structure my state is like:

module.exports = {
  global: {
    userId: '',
  },
  users: {
    '[id]': {
      name: '',
      email: '',
    },
  },
  messages: {
    '[id]': {
      toUser: '',
      fromUser: '',
      content: '',
    },
  },
  requests: {
    '[id]': {
      endpoint: '',
      success: false,
      failed: false,
    }
  }
}

My state tree has two types: fields and collections.

I have a global section with my flat fields of my state. They only exist on a single level. These fields don’t belong to a collection. Often they are identifiers.

Then I’ll have my collections. Each collection is an object of objects, stored by an identifier. The collections should be mostly flat data. If you have a collection that owns another collection, those should be two separate collections. And joined by an identifier. Think like how you would model relations in a SQL database, and that is the pattern you want to follow. If a collection doesn’t have a natural identifier, you’ll need to generate them with something like UUIDs.

Tal Kol has two great articles going further into state tree design. I would recommend reading those. As long as you are following the three rules above, you can model your state in any way that makes sense to you.

Your state schema can also serve as your default state. You only need a function that will empty out the collections into plain objects. Then you can use that to start up your Redux store in client.js and server.js. The more you integrate your state schema with your code, the more likely it will stay accurate.

function createDefaultState(schema) {
  return Object.keys(schema).reduce(
    (sum, key) => {
      sum[key] = isCollection(sum[key]) ? {} : schema[key]
      return sum
    },
    {}
  )
}

For writing your reducers, there’s been a growing consensus around the duck pattern and Flux Standard Actions. I would recommend using those. And your top level reducer can be Redux’s combineReducers.

I’ve worked on projects where every field and collection had its own unique set of actions. I’ve also worked on projects where the same set of actions were available on each field and collection, and all worked the same way. The consistent approach is easier to work with. For each of my “global”, flat fields, I will have two actions: set and reset. The set can work with any value, and reset will return the value back to the default. For my collections, I have seven action types: add one, add many, update one, update many, remove one, remove many, reset. And across collections, these all work exactly the same way. I like to have exactly those seven, so I always know what each collection can do without looking up my reducer.

// state/global.js
function globalReducer(state, action) {
  if (action.type === SET_USER_ID) {
    return { ...state, userId: action.payload }
  }
  if (action.type === RESET_USER_ID) {
    return { ...state, userId: schema.global.userId }
  }
  return state
}

// state/users.js
function usersReducer(state, action) {
  if (action.type === ADD_USER) {
    return { ...state, [action.payload.id]: action.payload }
  }
  // ... ADD_USERS, UPDATE_USER, UPDATE_USERS, ...
  // ... REMOVE_USER, REMOVE_USERS ...
  if (action.type === RESET_USERS) {
    return {}
  }
  return state
}

// state/index.js
const reducer = combineReducers({
  global,
  users,
  messages,
})

I make all my action creators the same way, using this sort of function:

const createAction = type => (payload, meta) =>
  ({ type, payload, meta })

const setUserId = createAction(SET_USER_ID)

dispatch(setUserId('abcd1234'))

If you choose to have consistent actions, you will see the vast boilerplate that Redux generates. I would wait a bit until the pattern is completely clear before trying to DRY it up, and even then a little at a time.

Side note: I haven’t had any positive experiences with immutable.js. Keeping mental track of plain versus immutable data was hard for a single contributor. And impossible for a team. If you’re thinking about that approach, immer seems a bit more reasonable.

Update/Edit 2018 Oct 10: Check out Redux Starter Kit and my Redux Schemad.

Test Data

Having good test data is the most efficient way to speed up development time for almost any project. I can’t recommend it enough. You can use test data for fast local development. You’ll also have a much easier time writing unit tests.

In my projects, I have a directory of data/ with two types: (a) fully written out test Redux states and (b) example responses from all the endpoints I’m going to use.

I will have at least four Redux test states.

An ‘ everything ’ state, that shows as many possible combinations of things as possible.
A null or zero state, which would be what the user first has when they arrive.
A loading state, that shows what the page looks like while services are loading, and
An error state, to show what happens when each service has an error.

As your project becomes more complex, you’ll likely need more states of each of these as well as other types of test states.

I like to have an easy way to load up each of test state and render the React tree on the test state. A good way to do this is using a query string parameter. This will make developing the React side of your project very fast. And unit testing your views as well.

I like having at least one sample of each endpoint I will use in the project. These samples will make formatting service responses and the corresponding unit tests very easy. You could also use your sample service responses to validate the real services match your expected data format. I’ve seen some make mock services based on the sample service responses as well. Mock services are not necessary for most projects, but mock services are an option.

Services

There’s lots of ways to make network requests. For one, straight up XMLHttpRequest might work for you. Otherwise, I’ve seen a growing consensus about the browser built-in, Promise-based window.fetch. If you need to server-side rendering, you can find a package to get fetch to also run on the server. (On async/await: Until there’s more browser support, it transpiles heavy… I wouldn’t use it in the browser yet.)

In my services directory, I would have either one file per family of endpoints (GET/POST/PUT/DELETE). Or I would do one file per endpoint, if there is one than one service with several endpoints.

For each endpoint, I have three functions. The top level is the actual service call. Which I name the function something like getMessage, listMessages, postMessage, putMessage, or deleteMessage. The format is {method}{noun}. The pattern here is:

Format state data to the parameters of the request (optional).
Update the state to record I’m making a request to this service.
Make the request, then…
If the request succeeded, format the response. Then dispatch an action to update the state with the formatted data from the response.
Either way, update the request in the state to record the request finished.

function getMessages(dispatch, getState) {
  const params = formatGetMessagesRequest(getState())
  const id = uuid4()
  dispatch(addRequest({ id, endpoint: GET_MESSAGES }))
  return fetch('/messages', params)
    .then(response => response.json())
    .then((response) => {
      const payload = formatGetMessagesResponse(response)
      dispatch(addMessages(payload))
      dispatch(updateRequest({ id, success: true }))
    })
    .catch(() => {
      dispatch(updateRequest({ id, failed: true }))
    })
}

Formatting the request and formatting the response can get verbose. So I will split those out into separate functions. For example:

I have a highest function, getMessage.
I have a function called formatGetMessageRequest. That takes the Redux state and other parameters. And returns an object of formatted parameters to make the request. The format I use is format{method}{noun}Request.
I have function called formatGetMessageResponse. That takes an API response. And returns the formatted response to match my state. That way I can dispatch actions to update my Redux state. The format I use is format{method}{noun}Response.

function formatGetMessagesRequest(state) {
  return {
    headers,
    body: {
      userId: get(state, 'global.userId'),
    },
  }
}

function formatGetMessagesResponse(response) {
  return get(response, 'messages', []).map(message => ({
    toUser: get(message, 'toUser', ''),
    fromUser: get(message, 'fromUser', ''),
    content: get(message, 'content', ''),
  }))
}

I’ve seen some use Normalizr to handle the formatting more systematically. I haven’t run into a project where I needed a tool to handle this, but that’s an option too.

Making service requests will likely be the messiest part of your project. As long as you expect service calls to be a little messy, you’ll be much happier.

Redux Middleware

So now you have a state tree, and you have service functions. But how do you know when to call your services? Or send analytics events?

Some developers use lifecycle methods in React — componentDidMount — with local component state. If you’ve already decided your project is complex enough to use Redux, I would recommend against doing this. React’s lifecycle methods and local component state are, well… state. And you already have your state in Redux. So if you use the lifecycle methods or local component state, you are violating the ‘single source of truth’ principle. As a result, its going to be much harder to determine what your project is doing or why you’re seeing the results you are seeing. Especially when you are on a team of people, if there’s state all over the place, its going to be difficult to get everything to work together.

The better solution is to use Redux to store all your necessary state for both the services and views. And to use Redux middleware to make state changes based on other state changes.

There’s some exceptions when you want to use React’s local component state and lifecycle methods:

You are building a reusable component for other projects to consume. Other projects may or may not use Redux. Even if a project is using Redux, they won’t use Redux the same way.
You have a demonstrated need to optimize the performance of a particular component.
Putting all the state for the particular component in Redux and Redux middleware would require lots of non-obvious work. I’ve seen this with tooltips and scroll behaviors, for example.

If you’re not in one of these three categories, I would stick to Redux and Redux middleware for all your state management needs.

So how do we write Redux middleware? Sometimes the direct approach is best. Redux’s authors chose middleware to use the currying pattern, so I recommend being familiar with currying.

There’s two primary ways to think about cascading state changes. One way is to listen to actions as we dispatch them, and call functions that match. redux-saga, a popular library, falls into this category.

const serviceMiddleware = store => next => action => {
  const result = next(action)
  if (action.type === GET_MESSAGES) {
    getMessages(store.dispatch, store.getState)
  }
  return result
}

The other approach is you dispatch a function that dispatches other actions for you. redux-thunk falls into this category. As do most other “async” libraries for Redux.

const thunkMiddleware = store => next => action => {
  if (typeof action === 'function') {
    return action(store.dispatch, store.getState)
  }
  return next(action)
}

dispatch((dispatch, getState) => {
  getMessages().then(() => {
    dispatch(setMessagesFetched(true))
  })
})

Lots of projects are using redux-thunk. When you dispatch a function instead of a plain object, the middleware calls the function. The function has access to store methods getState and dispatch. The access is what redux-thunk provides. Otherwise, it provides no guidance on how to make these state changes. For a single contributor, this is fine. For a team, this may be problematic.

I’ve seen both approaches — ‘listen then do’ and ‘do many’ — work. I have a preference for the “listen then do” approach. My reasons are:

You can keep almost all state management out of your views. Views only know how to dispatch plain actions and nothing else.
You end up with lots of little, easy to test functions instead of big functions that are harder to test.
It’s easier to keep your state changes in one place. As a result, its easier to see the flow of changes.

My own approach is to use my redux-listen library. That said, it isn’t popular by any means. I find redux-saga heavy for most projects, and redux-thunk provides too little guidance. You can also easily write your own Redux middleware that makes sense for your project. I can’t make a general recommendation here, other than it depends on the project.

Selectors

You now have a beautifully written state with no duplicate or derivative data. At some point, you need to take the data in your state and render out your view. That said, the format your state is in is probably not the format you want in your views.

The first thing developers often do is all the formatting in the view.

function Messages(state) {
  return (
    <div id="Messages">
      { Object.keys(state.messages).map(messageId => (
        <ul>
          <li>From: {state.users[state.messages[messageId].fromUser].name}</li>
          <li>{state.messages[messageId].content}</li>
        </ul>
      )) }
    </div>
  )
}

Soon though, the view has so much formatting and logic its hard to read or reason about. The next step is to use connect from react-redux and have a mapStateToProps function. mapStateToProps takes the full state and returns an object that will be the props for the view. You can move your data reformatting into mapStateToProps. Now you have an easy-to-read view and a separate formatting function.

function Messages({ messages }) {
  return (
    <div id="Messages">
      { messages.map(message => (
        <ul>
          <li>From: {fromUserName]}</li>
          <li>{content}</li>
        </ul>
      )) }
    </div>
  )
}

function mapStateToProps(state) {
  return {
    messages: Object.keys(state.messages).map(messageId => ({
      fromUserName: state.users[state.messages[messageId].fromUser].name,
      content: state.messages[messageId].content,
    }))
  }
}

However, the new problem you’ll have is you have many mapStateToProps functions that have duplicate formatting and logic. The solution to that problem is to use selectors. A selector is a function that takes a state and returns a derived value, e.g. state => value. You can combine selectors as well. Most developers will start all their selectors with the word get. You can call these selectors in your mapStateToProps and now you have a view with almost no formatting or logic in it.

function getMessagesNaive(state) {
  return get(state, 'messages', {})
}

function getMessagesArray(state) {
  const naive = getMessagesNaive(state)
  return Object.keys(naive).map(id => { ...naive[id], id })
}

function getMessages(state) {
  const users = getUsers(state)
  const messages = getMessagesArray(state)
  return messages.map(message => ({
    ...message,
    fromUserName: get(users, [message.fromUser, 'name'])
  }))
}

function mapStateToProps(state) {
  return {
    messages: getMessages(state),
  }
}

If you want, you can use the reselect library to help compose selectors together. You don’t have to use this library, but I recommend at minimum reading its README to grasp the pattern. Once you have lots of selectors, the library will be more useful to you.

Here’s the list of things you want to take out of your view and put in either mapStateToProps or a selector:

.length
+, -, *, \, %, **, in, <, >, ≤, ≥, ==, !=, ===, !==, &&, ||, ?:
Any function that determines logic, like RegExp.test
Any function that re-formats data, like map or filter
… and sometimes !

For functions that only do like number formatting or localization, I would leave those in the view.

The “sometimes !” comment is that you want to name any boolean values positive. So don’t use not, un-, hide, or anything negative in a boolean name. For example, !hideSection is confusing.

(The name selector is somewhat problematic, as a Redux/Redux selector is not the same as an HTML/CSS selector. It’s an overloaded term for browser development.)

I recommend grouping your selectors by route. So you’ll have a “base” file of selectors across routes. A selector file per route. And an “index” selector file that can combine selectors from the different routes.

Views: Pages, Containers, & Components

You’re now finally at the part of the application that uses React! If you’ve set up everything similarly so far, then this part is a breeze.

To repeat: I would recommend if you are using Redux, almost always write stateless functional components. (See the three exceptions in the “middleware” section.) If you are not in one of those three exceptions, then I would recommend writing simple, props-only functions across the board. That way, all your state information is in Redux and Redux middleware. You have a single source of truth. And your project becomes very predictable and easy to see why you’re getting the view you’re seeing at any time.

You can start with all your views in one, flat directory. As your project grows, I like to split them into three types of views:

Components : Simple, functional, props-only components. We haven’t attached to state or routes directly.
Containers : Components we’ve attached to state, but not directly to a route. (Note: If you have several route, you might not have any containers!)
Routes : Components we’ve attached to a route, and likely also the state directly. (If you are not using a router, then you’ll likely only have one route.)

Routes can consume any view. Containers can consume any containers and components. And components can only consume other components.

function Component(props) { return <div /> }

function Container(props) { return <div /> }
const mapStateToProps = state => ({})
export default connect(mapStateToProps)(Container)

function Route(props) { return <div /> }
const mapStateToProps = state => ({})
export default connect(mapStateToProps)(Route)
// ... a higher-level file
<Route path="/messages" component={Route} />

react-redux will come into play on containers and routes. You want to use connect and mapStateToProps to connect your state to your view. That keeps that stuff out of the view function itself. mapDispatchToProps will help you bind your action creators to your store. In some places, I’ve also had to create a mapPropsToProps to keep my selectors, logic, and formatting out the view.

JSX is the most controversial part of React. I would recommend using JSX. I don’t love the mixture of HTML and JS. It’s always felt weird to me, even years later. But most other projects are going to use JSX. And the tooling around JSX is spectacular. You get quite a bit for low effort by using JSX. I wish the authors of React chose a straight up JavaScript way to handle templating. But that decision is in the past now.

I’ve been on projects that attempted to use form libraries with either only React or with both React and Redux. Unless your form is basic, I would recommend against using a specialized form library. Form libraries often add lots of overhead for little benefits. I would stick to how the authors of React suggest doing controlled component forms . Wait to DRY up anything until you have a clear pattern of duplication.

Auxiliaries: Constants, Helpers, & Images

My projects have auxiliary directories in them: constants, helpers, and images. Sometimes there are more auxiliary directories too.

Constants help you to avoid magic strings. Constants encode the potential values your project will have. If you use constants instead of magic strings, eslint can help you out quite a bit when its time to refactor. Highly recommended.

Helpers are one off functions that don’t fit into other parts of the pattern. They are always specific to your project. I would recommend before creating any helper, check if there is a package in npm land you can use. Especially lodash. (A hint with lodash… you can install each function instead of the whole thing. That saves filesize in the browser.) I would try to keep to as few helpers as you can get away with, but nothing less.

For projects that need it, I’ll have a directory of images. Not much to say about images other than: optimize your images. You’ll get more performance improvement from optimizing images than anything else. Try using SVG, reducing the number of colors, and using ImageOptim. If you only have a small number of images that are vector-y, base64 encode them. This will save on the number of network requests. There’s plenty of articles about saving on size with images.

Tests

I’ve used Jasmine, Mocha/Chai/Sinon, and Jest. There’s more similar than not. If you already have one, I wouldn’t switch. That said, for new projects I recommend Jest. It’s the easiest to set up and get started with, and has everything you need for unit testing built in.

Jest’s killer feature is snapshot testing. Instead of having to assert every value of an object, or JSX nodes, you can write:

expect(result).toMatchSnapshot()

The first time you run that test, it will make a snapshot of the result. If you run the test later and it isn’t the same, you’ll see where the two snapshots differ. If you want to update the snapshot, you can do so on the command line easily.

I can’t recommend using snapshot testing enough. In particular, here’s some places you definitely want to use snapshot testing:

The results of your reducers in state/
The request and result formatters in services/ (If you have good test data, these can be one line tests!)
You can snapshot a mocked dispatch in your service calls. And it’ll record the arguments and returns each time we call the mocked function!
Anytime you use getState in your middleware/. And the mocking and snapshot-ing dispatch trick works here too!
Most of your selector/ tests can be a single line, if you use your test data and snapshot testing together.
Most of your view/ tests can also be a single line, if you use your test data and snapshot testing together. You can have one test for mapStateToProps. And another test for the view that combines it all together.

// Reducer test assertion
expect(
  messagesReducer(allState.messages, addMessage(payload))
).toMatchSnapshot()

// Request formatter assertion
expect(formatGetMessagesRequest(allState)).toMatchSnapshot()

// Response formatter assertion
expect(formatGetMessageResponse(getMessagesSample)).toMatchSnapshot()

// Service assertion
const dispatch = jest.fn()
const getState = () => allState
await getMessages(dispatch, getState)
expect(dispatch).toMatchSnapshot()

// Selector assertion
expect(getUsers(allState)).toMatchSnapshot()

// View assertions
expect(mapStateToProps(allState)).toMatchSnapshot()
expect(
  Messages(mapStateToProps(allState))
).toMatchSnapshot()

With simple functions, test data, and snapshot testing, your unit tests almost write themselves.

For other tests, you’ll want to use Jest’s mock capability. If you aren’t familiar with how mock internals, I have a video showing how they work. Jest’s system can handle about 90% of typical mocking needs. You can mock both individual functions and entire modules. Sometimes, the old school way works fine too:

test('updates correctly', () => {
  const prevLocation = window.location
  location = { href: 'https://example.com' }
  expect(something).toBe(another)
  window.location = prevLocation
})

One of the most frequent things to mock is time. Especially with snapshots, you want your tests to run deterministic.

Another strategy I use is to use call and apply to override the this context. This strategy gets around lots of otherwise tricky test cases.

test('componentDidMount', () => {
  const myThis = {}
  const result = Component.prototype.componentDidMount.call(myThis)
  expect(result).toMatchSnapshot()
})

Given snapshot testing, I’ve found getting to 100% unit test coverage very easy. If it were more difficult, I wouldn’t recommend 100% coverage for most projects. But in this combination, that goal is practical most of the time. If there’s ever a case you can’t cover, you can always put it in a file by itself and configure Jest to ignore that file.

You can do one of:

Create one big test directory at the root level.
Create a test directory in each main directory you have.
Create a test file sitting right next to each file.

I have a preference for 3. There’s twice as many files in each folder, but it’s easier to work with the files as you’re developing that way.

Wrap Up

I can’t cover everything needed to build a React/Redux project in one article. I’ve covered some of the patterns I’ve seen over the years. These are the ways I avoid the most common problems I’ve experienced. I haven’t covered to approach writing styles in this article. Or setting up the tooling required to support a production experience.

A personal note: I hope one day the browsers have efficient DOM element updating built. Something like…

rootElement.updateNodes(newDetachedDomTree)

This would cut the need for many of the tools I mention in this article. When a library gets popular enough, the browsers later get a way to do the same thing natively. I hope that happens here too.

To reiterate: Take what’s relevant to you and ignore the rest. Every project has its exceptions. You might not need JS, or React/Redux! This article is for 2018. If you are reading this any later, the community will have likely already changed some of this.

Thanks for reading. Feedback is welcome!

You can visit my side project Sagefy .

A Checklist to Make Your Own Learning Plan

Kevin Heis — Mon, 27 Aug 2018 00:00:00 +0000

I published my Eight Big Ideas of Learning article (also full version) about a month ago. The article attracted broad interest. Several people asked me how to act on those ideas.

If you’re an independent learner like me, you’re creating your own plan and also following that plan. It’s easy to feel overwhelmed. Let’s make it easier. Here’s some steps you can follow to create and follow your own research-backed learning plan!

The numbers in brackets refer to the sections in the Ideas article. Check that out if you need more clarification. You don’t need to do all of these things to learn effectively. Take what’s relevant and ignore the rest.

Before you start learning the subject

☐ Why do you want to learn this subject? Think autonomy, mastery and purpose. Write it down somewhere you will see regularly. [7.1]
☐ Learn the subject’s organization. Draw it out. You can try to find diagrams with Google image search, or use a Table of Contents from a popular book on the subject. [4.1]
☐ Determine a way to visualize your progress as you learn. Place it somewhere you will often see. [2.3]
☐ Assess what you already know. A few ideas: Find an online test, try some practice problems from a book, or ask an expert. [3.1]
☐ Use a tool to help you track what you need to do when. This could be a website, a To-do app, or even a paper checklist.
☐ Break down what you want to learn into small tasks of 30 minutes to an hour in duration. Set small and specific goals for each session. [2.1]
☐ Set goals with Bloom’s taxonomy in mind. Go beyond recognizing information. Add practice problems. Try miniature projects. Do comparison exercises. Try pattern exercises. Graph out concepts. Present arguments about what you’re learning. [2.2/6.2]
☐ Find a way to allow choices while you learn. You want choices for both content and format. Maybe on a given day you have the choice of one of three topics. Or one of three ways to learn the same topic, like a book, video, or activity. [5]
☐ Set your schedule with plenty of time for practice and spaced repetition. You should plan to revisit each unit at least two or three times, spaced out as you learn. [6.1/6.4]
☐ Find a way to get either expert or peer feedback as you learn. That could mean finding someone you already know to give weekly feedback. Or finding an online forum. This person or community can also help you stick to your plan. [8.1/8.3]

Before you start a learning session

☐ Find a quiet, calm, and clean learning environment. Remove distractions. It won’t be perfect, and that’s okay. [1.3]
☐ Only learn one thing at a time. If you are thinking about trying more than one thing at a time, don’t. If the session feels too large, break it down into smaller parts. [1.1/1.2]
☐ Review the organization of what you are about to learn. How does this fit into the subject? [4.1]
☐ What problem does this information or idea solve? Figure that out early. [7.2]
☐ Determine the main ideas of what you are about to learn. You could review section headers in a book, or a summary of the content. [4.2]
☐ Review related prior knowledge. What do you already know that relates to what you are about to learn? [3.3]
☐ Make a choice about the content or format before learning. [5]
☐ Limit your session to about 30 minutes to an hour. Take frequent breaks!

While you learn

☐ Tie back details with the main ideas. [4.2]
☐ As you learn, connect what you are learning to what you already know. [3.3]
☐ Stay in the middle of challenge. If too difficult, review what you already know and break down what you are learning. If too easy, skip ahead. [1.2/3.2]
☐ Scaffold. Solve easier problems before trying harder ones. Do isolated exercises before trying to create a full project. [3.2]
☐ Find examples. You could find a video demonstrating the idea. [7.3]
☐ Test your organizational knowledge. Draw out the relationships between the concepts. [4.3]
☐ Get feedback. Use your expert or peer to help make sure your learning is on task. Clear up any misconceptions and use this person to test what you’ve learned. [8.1]
☐ When you feel you’ve learned the content, update your progress visualization. [2.3]

To review weekly

☐ Revisit your reason for why you want to learn this subject. [7.1]
☐ Update your model of the organization of subject and your progress tracking. [4.1/2.3]

Ask meta-cognitive questions [6.3]… (There is no right answer to these. Just asking is enough.)

☐ Do I have a strategy? Is my strategy successful? [6.3]
☐ Am I focused while learning? Can I improve my focus? [1]
☐ Is this the right difficulty for me? [3]
☐ Are the choices I’m making effective? Do my choices have a limited number of options? Do my choices have clear default options? [5]
☐ Am I getting enough practice time? Am I spacing my practice? [6.1/6.4]
☐ Am I getting effective feedback? I am building trust with my expert/peer? [8.1/8.2]

Wrap Up

Learning is hard, and learning alone can be difficult. It can be difficult to stay motivated and keep going towards your goal. If you follow some of the ideas on this list, you can learn more easily, more deeply, and more quickly.

See the full article for more resources. Thanks for reading! Feedback is welcome.

If you’d like to know more about Sagefy, visit today.

Eight Big Ideas of Learning

Kevin Heis — Mon, 09 Jul 2018 00:00:00 +0000

I’ll admit it: I’m a huge nerd. One of my favorite subjects is learning how to learn. I’ve read and studied on learning science for many years. And I’ve built my own learning platform, Sagefy. That said, I’m no expert — enjoy this outsider’s take on the literature.

In this article, I’m going to sum up what I’ve learned about learning. There’s an entire field of study devoted to the subject, so I can’t include everything. But these eight points epitomize the research. (If this article is too long for you, check out my 10 minute TL;DR version.)

If you’re a learner too , this article is for you. Maybe you’re a student, a teacher, or a lifelong learner like me. Education researchers usually write these from the perspective of a classroom teacher. I am writing from the perspective of an individual, independent learner. These ideas can apply to a classroom, online learning, and everyday life lessons. And come back and review these ideas every now and again… you might find a new perspective you didn’t expect. I always do.

I believe in evidence. Research isn’t perfect. But research is better than the alternative. We all develop theories of learning throughout schooling and working. But not all these theories have evidence to back them up. Take “learning styles” — the theory that each person has a type of learning, like auditory or visual. The idea is pervasive in institutional education. Yet, the theory of “learning styles” does not have evidence to support it. Feedback from learners about their preferences is another illusion. Instead, we need to focus on learning results.

It’s not always easy to pull research into applicable techniques. Articles like this attempt to bridge the gap. These ideas are not new or unique. I am not the first person to say any of these things. While this specific list and order is unique, every idea in this article comes from other sources. I will link where possible.

If my background interests you: Sagefy is the project I’m building on these ideas. Sagefy is an open-content adaptive learning system. Open-content means anyone can create and update learning content, like Wikipedia. Adaptive learning means the content changes based on the learner’s prior knowledge. The combination means anyone can learn almost anything, regardless of their prior knowledge. To learn more about Sagefy, check out this in-depth article.

The greatest opportunity for technology is human learning. We accelerate our progress by using our tools to accelerate our knowledge. We must avoid technology for technology’s sake alone. Technological advances often do not living up to their possibilities. The history of ed-tech shows technology deviating from learning principles. We avoid unnecessary technological clutter by focusing on core principles. I want this article to be a reference for myself as I build and grow Sagefy.

Idea 1: Do One Thing at a Time

“ Do one thing at a time” is a long way to say: focus. Remove distractions. Drop unneeded interfaces. Silence background chatter. Don’t multitask: it doesn’t work. Only focus on one lesson of one subject at a time. When we focus, we get more out of the energy we put into learning.

Attempting to learn three new ideas at the same time to solve one problem is going to slow you down. Instead, isolate and learn each idea, and then integrate the ideas to solve the problem.

1.1 Account for the limits of working memory.

Our brain has about 86 billion neurons. Our long term memory appears to be limitless. Our reality is that we cannot remember everything in our long-term memory at one time. What we remember in the current moment, we call working memory.

George Miller in 1956 found working memory limited to seven items, plus or minus two. Decades later, researchers found our brain limits working memory to four items. This rule applies for anything beyond rote memorization. See Broadbent in 1975, Baddeley in 1994, and Cowen in 2001. IQ measurements generally do not impact working memory limits.

We learn by connecting new information to previous known information. (As we’ll come back to in an upcoming section (3.3).) Your current knowledge takes one to three items. That means you likely only have one or two items left for new knowledge. Much of the rest of the points in this article deal with the limits of working memory.

In summary, be realistic about how much information you can put into your working memory. It’s less than you expect.

1.2 Do one activity at a time. Avoid multitasking.

Humans are terrible at multitasking. One study by Clark, Ayres and Sweller in 2005 asked learners to learn both spreadsheet software and related mathematics at the same time. The learners felt overwhelmed. Another 2001 study by Mayer and Chandler found learners overwhelmed learning an entire interface at once. Learners were more successful learning the interface sequentially.

An entire field of study, capacity theory, has explored the idea of how much one can learn at the same time. Many researchers use the Wisconsin Card Sorting Testing to show we can’t multitask while learning something new.

The gist is that trying to learn more than one thing at a time is counterproductive. Instead, it’s much better to try to learn one thing at a time instead of many things at once. Learn one thing, take a break, learn the next thing, take a break, and so on. If you’re feeling overwhelmed, break down what you’re learning into different steps. Then, bring it all together at the end.

1.3 Cut noise: distractions, extra choices, tangents, extra inputs…

A 1996 Mayer et al study found summarizing a lesson, and reducing information, increased learning. A 1998 Harp and Mayer study found that adding interesting bits of information reduced learning. A 2001 Mayer study found increasing the amount of information in a lesson decreased retention. A 2006 Butcher study found simple diagrams work better for learning than realistic diagrams. Finally, a 2007 Mason et al study found wandering minds were not as capable on performing on tasks.

Noise can come in many forms. Noise detracts from learning. There’s auditory or visual noise. There’s noise as extra choices in the learning interface. And there’s noise in the form of tangents of information. For example, one of my goals for Sagefy is to reduce interface noise to keep the focus on learning. Find a quiet, comfortable place to learn where nothing will distract you. Disable social media, put the phone on quiet, close the browser tabs. Get rid of extraneous thoughts, worries, and planning. Allow yourself to focus on learning one thing at a time.

Idea 2: Set & Adhere to Goals

Define small and achievable goals. Then as you learn, check against those goals. Setting specific goals gives the opportunity for frequent wins, which keeps you learning.

If your goal is too large or ill-defined, you’ll have difficulty focusing and lose motivation. You’ll deal with the constant question, “what’s next?” and spend more time on figuring out what to do than actually learning. By setting small and achievable goals, you keep learning.

2.1 Define small and achievable goals.

There are many goal frameworks such as SMART. What the frameworks all seem to share is making goals that are both small and achievable.

Goals and learning are one of the most studied relationships in educational science. A pivotal study in 1979 by Rothkopf and Billington asked students to set small goals before they started reading. This improved reading comprehension. A 2000 paper by Deci and Ryan finds goals are a fundamental psychological need. A summary article in 2002 by Locke and Latham goes into the long history of the research of setting goals in learning. The researchers showed setting goals — specific, challenging, and achievable — improves learning. A 2006 paper by Kivetz, Urminsky, and Zheng found setting small goals increased desired behavior. A 2010 paper by Koo and Fishbach found focusing on remaining tasks — not completed tasks — delivered higher results. In Ericsson’s deliberate practice model, deliberate practice requires frequently setting goals. The other aspects are focus (1.3), feedback (8.1), and appropriate level of challenge (3.2).

You should start each learning session with a small and specific goal. For example, learning all of calculus is huge and not achievable in one session. Instead, set a goal of solving 20 practice problems for the derivative power rule. That is both small and achievable. Often, the more specific the goal, the more achievable the goal is. Losing motivation while learning is common. Setting small and achievable goals helps us to stay motivated. And setting goals helps us to also live up to Idea 1: Do One Thing at a Time by helping us to determine our focus.

2.2 Set goals for both knowledge and skills.

Bloom’s 1956 taxonomy defines six types of learning activities. Knowledge, comprehension, application, analysis, synthesis, and evaluation. (Also see Krathwohl’s 2002 revision.) If we only set goals for knowledge, we miss the opportunity to develop the other parts. For example, a 1989 Chi et al study found that stronger learners explain what they’ve learned in their own words.

One of the most common sayings is learning has no value unless you can apply what you’ve learned. So we need to make sure the goals we set go beyond memorization of facts. Challenge yourself to relate data points to each other. Work on projects that apply the knowledge. Do comparison exercises. Spend time identifying patterns in the knowledge. Present arguments based on what you’ve learned. Going beyond knowledge is important. (We’ll revisit this aspect further in section 6.2.)

2.3 Habitually review your progress against your goals.

Daniel Pink’s Drive shows intrinsic motivation is more powerful than extrinsic motivation. Extrinsic motivation is reward and punishment. Pink identifies three main principles of intrinsic motivation. Autonomy (see Idea 5), mastery (see Idea 6), and purpose (see Idea 7). Checking our progress against our goals grows our sense of mastery. That in turn builds intrinsic motivation.

In Sagefy, as you learn you get to see your progress of how many units you’ve mastered in the subject. And as you go through learning activities, a simple bar shows your progress to mastering the unit.

As you learn, create a way to visualize your progress against your goals. That reminder can help keep you motivated towards learning.

Idea 3: Adapt to Prior Knowledge

The strongest predictor of how much we will learn is what we already know. Prior knowledge is how much we know going into a learning experience. Adapt the learning experience to prior knowledge. When the learning activity pushes us to the right level of challenge, we make the most of our time learning.

If the content is too easy, we get bored. If the content is too difficult, we get frustrated. Our understanding becomes vague. Either way, we lose motivation. By staying in the in-between zone, we get the most of our learning effort and keep with it for longer.

3.1 Assess prior knowledge first. Focus on areas of weakness.

Dochy, Segers, and Buehl in 1999 found 81% of the difference between learning rates in different learners is prior knowledge. In Hattie’s massive ranking of learning factors, prior knowledge has an effect size of 0.94. Prior knowledge is also the underlying effect of two even stronger factors. Prior knowledge has the largest impact on how much we learn.

Formative assessment is the idea that we should start by testing our knowledge first. Then, we continue to use assessment to learn. Summative assessment is only at the end. A NY Times article highlights the value of formative assessments. In an influential 1972 Bransford and Johnson article, formative assessment activated prior knowledge. The result was that formative assessment is a learning tool itself. A Wininger study tried using assessment both before and during learning activities. This elevated learner comprehension.

We often decide to pick a book or a course for a topic without considering what we already know first. Industrialized education unfortunately suggests all learners are starting from the same place. If you want to learn how to play piano without knowing how to read sheet music, you are in for a difficult journey. Instead, find a means to assess what you already know— and what you need to work on — before you start learning. And, use assessment as you learn to identify areas of weakness and build up in those areas first.

3.2 Stay in the middle and adjust the challenge. Scaffold.

Mihaly Csikszentmihalyi’s most famous work is Flow. Flow is a psychological concept of being completely engaged with an activity. We are in “learning flow” when our level of skill and the level of challenge match. Sometimes our prior skill level and the level of challenge don’t align. Then we are anxious or bored. John Caroll’s Minimalism framework also promotes matching the challenge to the learner’s skill level. We can trace this idea back to Atkinson’s 1957 article.

One of the ways to help match prior knowledge and challenge level is to use scaffolding. Scaffolding is the idea of putting on training wheels. At first, provide lots of learning support. Then, remove those supports as the learner’s skill develops. A review by Reiser shows scaffolding is an old and effective technique.

In Sagefy, I measure and update how strong the learner knows the material with each activity. I show the learner activities to choose from and engage with based on those measurements.

We want to avoid learning experiences that are too easy or too difficult. Too easy, and we get bored and waste time and effort on things we already know. Too difficult, and we get frustrated and start to shut down. We want to stay in the middle. Sometimes that means breaking down what we are learning into smaller parts first. That might mean reviewing prerequisite knowledge. We can also scaffold ourselves. Start with simple applications of knowledge, and slowly increase the complexity.

3.3 Connect new information with existing knowledge.

In 1980, Gick and Holyoak asked learners to make analogies connecting their prior knowledge to new knowledge. This improved recall. A 1991 study by Martin and Pressley had students explain how what they already knew applies to new information. This had a dramatic impact on learning. A 1992 review of prior knowledge by Pressley et al looks into the history of prior knowledge. The authors conclude we learn by connecting new information with prior knowledge. A 2004 chapter by Chance, DelMas, and Garfield arrives at similar conclusions.

Before learning something new, start by reviewing related information you already know. Otherwise, you lower the chance to associate those two pieces of knowledge. You reduce your ability to recall the new information when relevant. For example: you are going into a lesson about choosing effective typefaces. You could start by reviewing the different categories of typefaces. Connecting what you already know to what you want to learn is one of the best ways to accelerate your own learning.

Idea 4: Build the Graph

Our memory system works by forming relationships between data. Mastery is the result of a large, deeply-connected graph. We link new information with information we already know. When we focus on organizational knowledge, as we learn we have mental places to put the knowledge.

When we don’t know how experts organize the knowledge before we dig in, we don’t connect the knowledge. Learning becomes mechanical. By learning how to organize the information first, we can quickly connect information. We can use our prior knowledge more. And we get more out of each piece of the graph.

4.1 Learn how to organize the subject before getting specific.

One heavily cited work in educational research is DeGroot’s 1965 “Thought and Choice in Chess.” In this work, DeGroot shows the differences between beginner and expert chess players. Chase and Simon’s 1973 study expands on that work. In that study, beginner and expert chess players differ in organizational knowledge. A 1969 study by Bower et al presented hierarchies of words to learners instead of unorganized lists. This elevated recall. In Reif and Eylon’s 1984 study of a physics lesson, researchers presented organization first. This improved test performance. Another Chi et al study found similar conclusions.

Learn the subject’s organization before you begin to learn something new. That could mean reviewing the Table of Contents in a book before reading it. Or reading the section headings before reading the full text. Review maps and diagrams of how to organize the information. This organizational task can help give structure to your learning plan. Organizational knowledge also helps you to connect appropriate prior knowledge (3.3). And set goals (2.1). And as you learn, you’ll know how the information relates to other areas within the subject.

4.2 Work top-down.

Top-down learning means learning the organization and main ideas first. Then dive into each of those solo. Get deeper as you move further along. Bottom-up learning means learning each component in isolation, then combining the knowledge only at the end.

A 2002 Sun and Zhang study tried top-down processing and bottom-up processing on the “Tower of Hanoi” problem. The top-down approach was more effective. A 2006 article by Lovrich finds use cases for both forms, but generally top-down is more effective. Finally, a 2008 study by Lim, Reiser, and Olina surveyed the whole task first, then drilled down. That was more effective than teaching individual parts then combining at the end.

We know that the brain limits working memory (section 1.1). One strategy to deal with this limit is chunking. Chunking means to group several similar items together. Chunking is one strategy to add a top-down workflow to your learning. A 1980 study from Ericsson, Chase, and Faloon shows chunking nears expert performance for memorizing numbers.

As you learn, try to figure out what the main idea of the lesson is. Then, try to find ways to tie back in each point with the main idea.

4.3 Test your organizational knowledge.

A 1977 Hinsley, Hayes, and Simon study asked students to categorize algebra problems. When students had correct categories, they more often arrived at the correct answer. Having students practice categorizing problems improved performance. A Chi & VahLehn 1991 asked learners to explain and include organizational knowledge. The explanations with organizational knowledge were more effective.

As you learn, find ways to test your organizational knowledge. For example: you could pick out a passage in a text. Then, figure out where that knowledge fits into your schema of the subject. Or you could use an existing diagram and fill in the gaps. You could also try concept mapping; drawing out a map to structure the subject. When you understand the organization, you learn faster and keep the knowledge.

Idea 5: Empower Choice

Autonomy is one of the “three” major intrinsic motivators. Choice fuels autonomy. When we take ownership of our learning through choice, we are more motivated to stick with it. For example, choice can be choosing between one of three units to learn next. Or one of five different formats to learn the same lesson.

If someone forces learning on us, we resent the requirement. Our focus becomes our resentment, not learning. Too many choices can break down our ability to learn too. We focus on making choices instead of learning. And most of us are terrible at making good choices when there are a vast number of options. Instead, the right amount of choice as we learn keeps us motivated and focused on learning.

5.1 Create a system that often allows for choices, in both content and format.

Sheena Iyengar is a prominent researcher in the field of the psychology of choice. She is the author of “The Art of Choosing.” In the book, Iyengar accounts for our fundamental desire for control. The sense of having control can lead to some irrational behavior. Choices can also be motivating as a form of ownership. In a 1989 chapter in “Goal Concepts…,” Bandura finds goals we set ourselves are stronger than goals others set for us.

In Schankenberg’s 1997 dissertation, she found learners practiced more when allowed to make choices. The extra practice improved outcomes (also see the more often cited 2000 republishing). She also found learners reported higher motivation and satisfaction from the course. A 2010 Tabbers and de Koeijer study also found when learners had choice, learners spent more time on instructional material. And as a result had higher transfer.

As you create a learning plan for yourself or others, think about ways to include simple choices. For example, you can choose one of three activities for the same information. Or it could be the format is the same, but you can choose which of four things you would like to learn next. Your learning plan for yourself might look more like a graph instead of a list. That way, given the day you can choose which part to work on next instead of feeling forced into the next step.

5.2 Cut unnecessary choices and reduce the number of options.

Choice can overwhelm. A 2000 study by Iyengar and Lepper looked at the number of options per choice. A limited number of choices improved results and motivation.

Learner choice is likely the most controversial subject in educational research. A paper by Lunts reviews the research. And finds mixed results based on context.

Choice is powerful. Often too powerful. We need to be careful to not overwhelm ourselves with too much choice. Again, the purpose of choice is to create a sense of autonomy and control to drive motivation. The purpose of choice in learning is not optimization.

5.3 Create a recommended default choice.

A learner’s own perception of ability is often inaccurate. A 2010 Zabrucky review highlights this common result in research. Jee, Wiley, and Griffin found learner assessment of ability low for both novices and experts. In a 2004 study by Eva, Cunnington, Reiter, and Norman, past performance predicts results more than learner predictions.

Avoid putting yourself or another learner in a position to make a poor choice. Avoid choices that require learner self-assessment to make an informed choice. One way to avoid these situations is to offer choice with a small number of options. And present a default choice if the learner wants to move on without choosing. You want choice to be about the feeling of control and autonomy, not about optimization.

Idea 6: Dive Deep

Rote memorization is not enough. We must apply knowledge to create results. When we learn not only what, but also how and why, we empower ourselves to use the knowledge we gain. Asking the questions “how” and “why” deepens what we know.

Rote memorization is boring. No one enjoys learning that way. We forget knowledge we learn by memorizing. Instead, we need to focus on how the knowledge works and why the knowledge is relevant. And these are separate goals than “the what”! Diving deep means learning is more interesting and we keep the knowledge for longer. Diving deep means we can apply what we learn and make a real impact.

6.1 Practice.

Getting enough practice is important for skill development. That may sound obvious. But, in school we get this idea that we will learn something by only hearing a teacher recite the information. We also have strong misconceptions about how much practice we need.

One interesting idea here is the Power Law of Practice. Decades of studies have found a power correlation between practice and skill efficiency. In “How I Rewired My Brain to Become Fluent in Math”, Barbara Oakley discusses the importance of mastering the fundamentals. She does this through deep practice before moving on to more difficult subjects. A 1996 study by Sloboda et al found that the amount of practice was an even stronger indicator of results than prior ability. A 1998 study by Healy et al found practice needs to have variety that resembles real application. A 2006 study by Keehner et al found practice reduces ability differences in learners over time. The Foreign Language Institute believes it takes 600–2,200 hours of practice to reach moderate skill with a language. The Language Testing Institute reports a similar range.

You need to build more practice in your learning plan than you might guess.

6.2 Go beyond rote: explain, apply, analyze, and synthesize.

As I mentioned in 2.2, Bloom’s taxonomy identified six core skills. Knowledge, comprehension, application, analysis, synthesis, and evaluation. Craik and Lockhart’s 1972 memory framework shows we need a variety of tasks to build applicable knowledge. A 1997 Schnieder and Shiffrin paper looks at this as well. The authors find what makes fixed and flexible expertise different is the variation of practice. Koedinger & Anderson’s 1998 paper shows learns with higher and performance when practicing a variety of tasks. We find similar messages in McKeough, Lupart, & Marini’s 1995 “Teaching for transfer”.

As you create and update your learning plan, make sure to include a variety of practice. Include exercises that need not only recognition or recall of information. Include exercises that have you to explain, apply, analyze, and synthesize knowledge.

6.3 Build metacognition.

Metacognition — in context of learning — is the ability to evaluate your own learning strategies… and adapt your strategies as you go. Metacognition is a major topic in educational research. There’s way more than I will include in this article, but I’m going to touch on the basics.

Flavell’s 1976 “Metacognitive aspects of problem solving” is an early approach of the subject. He identifies motivation and regulation as key metacognitive capabilities. Several studies have confirmed learners with higher metacognitive ability learn better. A few examples: Palincsar & Brown 1984, Scardamalia Bereiter & Steinbach 1984, and Sengul & Katranci 2012.

As you are learning, every few hours or so, ask yourself a few questions:

Am I focused on learning? How could I improve my focus?
What should I be paying attention to?
Have I done something similar before? Have I experienced something related before?
How well do I understand this? How difficult is this for me right now?
Do I understand how to organize this knowledge?
Do I have a strategy? Is my strategy successful? How many different approaches did I try?

Ask these sorts of questions often. These questions improve your metacognitive ability. And lead you to stronger learning results.

6.4 Space your practice.

One of the oldest studies in educational science is Ebbinghaus, 1885. Ebbinghaus generalizes retention of knowledge to a curve. The stronger the memory, the longer we keep the memory. And the more spaced repetitions of the event, the stronger the memory. Baddeley & Longman in a 1978 found typists who spread out their practice had stronger outcomes. Bahrick and Phelps found in 1987 that Spanish learners who spread out their practice had much stronger retention. Caple’s 1996 paper shows spaced practice improves learning and retention.

Spread out your practice rather than cramming. As the memory becomes stronger, you can go longer times without review. Create a system for yourself, some sort of reminder, to review your knowledge as you learn.

Idea 7: Make It Real

Purpose is another major intrinsic motivator. We need to know not only why the knowledge is important to the subject. We also need to know why the knowledge is important to us. Real-life connection and examples gives the knowledge purpose.

When we remove knowledge from our experiences, we can’t see why we should learn. Instead, review “why should I care?” With this, we can keep our motivation going. And make it easier to apply the knowledge later.

7.1 Check your own reasons for learning.

Purpose is one of Pink’s three intrinsic motivators. A 2004 Robbins et al study looks at learning goals. The study finds goals aligned with intrinsic motivations improve of college outcomes. A 2004 Vansteenkiste et al study also says goals with underlying intrinsic motivation are stronger. A 2002 review of research by Eccles and Wigfield comes to again similar conclusions.

Before you learn, write down why you want to learn the subject. Focus on intrinsic motivation. Yes, extrinsic goals — like getting a job — can be a meaningful goal. But also focus on what that means for you: autonomy, mastery, and purpose. Does getting that promotion mean more autonomy in your day-to-day work? Is that course going to help you help others in some way? Go for deeper purposes.

As you learn, remind yourself of that goal every now and again. And make sure you connect the day’s goal with the bigger picture. It’ll keep you motivated to stay learning!

7.2 Connect material with real-life problems.

Think: problem → solution.

One early theory in learning science is Dewey’s 1916 Democracy and Education. His main thesis is that the mind does not form on its own, but as a social function. This thesis leads to the idea of learning by doing — when we learn in a real context, we deeply learn. When we see what we are learning is real we connect with the material deeper. A 2000 Köhler et al study finds images of concrete nouns stimulated more than textual forms. Problem-based learning is common in health sciences. PBL means focusing on the problem, and learn to find the solution. A 2016 review by Jin and Bridges finds mixed results in this new area of learning research. But learner motivation is higher when we connect the material to real problems. A 2013 Wired article and 2014 NYTimes article shares this idea in story form.

As you learn, try to connect what you are learning to something real to you. An easy way to do this is to ask, “What problem does this solve?” The problem is the context.

7.3 Learn from examples. Use audio, images, video, and interactive.

Listening to lectures and reading textbooks is common. But we know from the research that an example is so much clearer. Examples can be in visual, auditory, or interactive form. A 1985 Sweller and Cooper experiment looked at algebra student performance. The study found swapping some textual description with worked examples is more effective. A 1996 Marcus et al study finds examples lead to stronger understanding of instructions than text alone. Even describing the visual aspects can lead to improved results, as found in a 1997 Kealy et al study. A 1980 Clement study found diagrams to be an effective way to combat misconceptions. A 1997 review by Robertson finds most learners imitate examples as an early strategy. Diagrams are powerful, as a 1987 Larkin and Simon review finds. A 2000 review by Atkinson et al provides guidelines for using examples in learning.

Include examples in a variety of formats in your learning plan. If you get stuck, search for an example using an image or video search engine. Or create your own diagram to help with your own understanding!

Idea 8: Learn Together

When we learn with other people, we can accelerate both our own learning and others’ learning. We can share ideas. Challenge each other’s misunderstandings. And provide and gain feedback.

When we learn alone, we get stuck more often. With no feedback, we miss the opportunity to correct our weaknesses. Without peer accountability, we can let the rest of life take priority over learning. Learning together changes what we learn, when we learn, and how often we learn.

One our key learning drivers is mimicking successful behaviors of others. In Galef and Wigmore’s 1983 article, rats imitated the successful behavior of demonstrator rats. We’re social creatures and learn together.

Not all peer learning and group learning is successful. We need to be careful in how we interact as we learn together to keep things positive and constructive.

8.1 Invite and offer feedback.

Feedback is a powerful and heavily researched topic in educational science. Scientists studied the nuances of asking for and inviting feedback.

Anderson, Corbett, and Conrad in a 1989 study studied computer programming teaching software. The authors found feedback kept learners in the software longer. And improved learning results. Hattie places feedback high several times, with a direct effect size of 0.7. Hattie and Temperly’s own review of feedback in 2007 gets into the nuance. But, the authors find regular and timely feedback improves motivation and learning outcomes. The more focused and task-specific the feedback is, the better the results. Shute’s 2007 research-based guidelines on feedback find similar conclusions.

Feedback isn’t automatic. Unfortunately as humans we’re prone to the bystander effect. We believe someone else will take action. And the more people that are available the less likely we are to take action. So we shouldn’t expect others to provide feedback. We need to ask for it.

In summary, we need to both ask for and offer feedback. Feedback should be timely. Feedback should be as tailored as possible. When you are learning, see if you can find someone who can offer you feedback as you learn. And you can learn as much by offering feedback to others too.

8.2 Be honest and forthcoming.

In Ford’s 1992 “Motivating Humans”, Ford describes supportive and unsupportive environments. Learners have stronger outcomes in supportive environments. Beck et al’s 1996 “Questioning the Author” informed learners about teaching strategy. When learners knew of the strategy engagement increased. In a 2010 Ritchie and Thorkildsen study, researchers compared learners who knew about the teaching strategy to those who didn’t. The researchers found when learners knew, the learners had higher performance.

We learn better when we are honest and forthcoming with each other. When others are honest with us, our trust in them grows. We’re more willing to let down our shield and let in what they know.

8.3 Hold each other accountable.

A 2014 Russell study found accountable peer groups performed stronger than self-accountability. An article in Harvard Business Review by Grenny looked at teams that rely on peer accountability. …rather than accountability coming from leaders. The peer accountability teams have higher performance.

When we hold each other accountable for learning, we’re more likely to stick with it and finish our goals. One strategy you can try is to find a ‘study buddy’ and have regular, recurring check-ups.

8.4 Build consensus.

In a 2002 De Grave et al study, researchers viewed groups with members reporting “unequal participation” and “lack of cohesion.” Those groups did not deliver learning outcomes. A 2012 Walton et al study asked members to believe another group member shared the same birthday. This improved learning outcomes.

We are social creatures, and we are social learners. When learning with others, consensus can drive learning results. The sense of belonging and alignment can help us to learn faster.

Wrap Up

I wrote this article from the perspective of the individual, independent learner. But these ideas apply to the classroom and to training as well!

Think about ways to create an environment where these ideas can flourish. You could even print out this list and hang it on a wall or another place you’ll see it often.

Update: Wondering how to act on these ideas? Check out my “Checklist” article!

Thanks for reading! Feedback on this article is welcome. I intend this article to be a living document. Did you find some new research to support or contradict one of these ideas? Or something could be clearer or more applicable? Let me know by writing a response!

Links for Further Reading

How Learning Works: Seven Research-Based Principles by Ambrose, Bridges, DiPietro, Lovett, & Norman
Building Expertise: Cognitive Methods for Training and Performance by Ruth Clark
Learning How to Learn by Barbara Oakley (Also see her Top 10 List)
How People Learn by Bransford, Brown, Cocking, & Pellegrino
Peak: Secrets from the New Science of Expertise by Ericsson and Pool
Drive by Pink
Kaplan Checklist
Pearson Learning Design Principles

If you’d like to know more about Sagefy, visit today.

Eight Big Ideas of Learning [TL;DR Edition]

Kevin Heis — Mon, 09 Jul 2018 00:00:00 +0000

This article is a summary of another article I wrote. Check it out if you want more details!

One of my favorite subjects is learning how to learn. I’ve read and studied on learning science for many years. That said, I’m no expert — enjoy this outsider’s take on the literature. In this article, I sum up what I’ve learned about learning in eight points. If you’re a learner too , this article is for you.

If my background interests you: Sagefy is the project I’m building on these ideas.

Idea 1: Do One Thing at a Time

1.1 Account for the limits of working memory.

What we remember in the current moment, we call working memory. Our brain limits working memory to four items. We learn by connecting new information to previous known information. In summary, be realistic about how much information you can put into your working memory. It’s less than you expect.

1.2 Do one activity at a time. Avoid multitasking.

Humans are terrible at multitasking. Trying to learn more than one thing at a time is counterproductive. Instead, it’s much better to try to learn one thing at a time instead of many things at once. If you’re feeling overwhelmed, break down what you’re learning into different steps. Then, bring it all together at the end.

1.3 Cut noise: distractions, extra choices, tangents, extra inputs…

Noise can come in many forms. Noise detracts from learning. Find a quiet, comfortable place to learn where nothing will distract you. Disable social media, put the phone on quiet, close the browser tabs. Get rid of extraneous thoughts, worries, and planning. Allow yourself to focus on learning one thing at a time.

Idea 2: Set & Adhere to Goals

Define small and achievable goals. Then as you learn, check against those goals. Setting specific goals gives the opportunity for frequent wins, which keeps you learning. If your goal is too large or ill-defined, you’ll deal with the constant question, “what’s next?” and spend more time on figuring out what to do than actually learning. By setting small and achievable goals, you keep learning.

2.1 Define small and achievable goals.

You should start each learning session with a small and specific goal. Often, the more specific the goal, the more achievable the goal is. Losing motivation while learning is common. Setting small and achievable goals helps us to stay motivated.

2.2 Set goals for both knowledge and skills.

2.3 Habitually review your progress against your goals.

Checking our progress against our goals grows our sense of mastery , one of the three powerful intrinsic motivators. That in turn builds intrinsic motivation. As you learn, create a way to visualize your progress against your goals. That reminder can help keep you motivated towards learning.

Idea 3: Adapt to Prior Knowledge

The strongest predictor of how much we will learn is what we already know. Prior knowledge is how much we know going into a learning experience. Adapt the learning experience to prior knowledge. If the content is too easy, we get bored. If the content is too difficult, we get frustrated. By staying in the in-between zone, we get the most of our learning effort and keep with it for longer.

3.1 Assess prior knowledge first. Focus on areas of weakness.

We often decide to pick a book or a course for a topic without considering what we already know first. If you want to learn how to play piano without knowing how to read sheet music, you are in for a difficult journey. Instead, find a means to assess what you already know— and what you need to work on — before you start learning. And, use assessment as you learn to identify areas of weakness and build up in those areas first.

3.2 Stay in the middle and adjust the challenge. Scaffold.

We want to avoid learning experiences that are too easy or too difficult. Too easy, and we get bored and waste time and effort on things we already know. Too difficult, and we get frustrated and start to shut down. We want to stay in the middle. Sometimes that means breaking down what we are learning into smaller parts first. That might mean reviewing prerequisite knowledge. Start with simple, and slowly increase the complexity.

3.3 Connect new information with existing knowledge.

Before learning something new, start by reviewing related information you already know. For example: you are going into a lesson about choosing effective typefaces. You could start by reviewing the different categories of typefaces. Connect what you already know to what you want to learn.

Idea 4: Build the Graph

Our memory system works by forming relationships between data. Mastery is the result of a large, deeply-connected graph. We link new information with information we already know. By learning how to organize the information first, we can quickly connect information. We can use our prior knowledge more. And we get more out of each piece of the graph.

4.1 Learn how to organize the subject before getting specific.

4.2 Work top-down.

Learn the organization and main ideas first. Then dive into each of those solo. Get deeper as you move further along. As you learn, try to figure out what the main idea of the lesson is. Then, try to find ways to tie back in each point with the main idea.

4.3 Test your organizational knowledge.

As you learn, find ways to test your organizational knowledge. Try concept mapping; drawing out a map to structure the subject. When you understand the organization, you learn faster and keep the knowledge.

Idea 5: Empower Choice

Autonomy is one of the “three” major intrinsic motivators. Choice fuels autonomy. When we take ownership of our learning through choice, we are more motivated to stick with it. If someone forces learning on us, we resent the requirement. Our focus becomes our resentment, not learning. Too many choices can break down our ability to learn too. Instead, the right amount of choice as we learn keeps us motivated and focused on learning.

5.1 Create a system that often allows for choices, in both content and format.

As you create a learning plan for yourself or others, think about ways to include simple choices. For example, you can choose one of three activities for the same information. Your learning plan for yourself might look more like a graph instead of a list. That way, given the day you can choose which part to work on next instead of feeling forced into the next step.

5.2 Cut unnecessary choices and reduce the number of options.

Choice is powerful. Often too powerful. We need to be careful to not overwhelm ourselves with too much choice. The purpose of choice is to create a sense of autonomy and control to drive motivation. The purpose of choice in learning is not optimization.

5.3 Create a recommended default choice.

A learner’s own perception of ability is often inaccurate. Avoid putting yourself or another learner in a position to make a poor choice. Avoid choices that require learner self-assessment to make an informed choice. Present a default choice if the learner wants to move on without choosing. You want choice to be about the feeling of control and autonomy, not about optimization.

Idea 6: Dive Deep

Rote memorization is boring. No one enjoys learning that way. We must apply knowledge to create results. When we learn not only what, but also how and why, we empower ourselves to use the knowledge we gain. Diving deep means learning is more interesting, we can apply what we learn and make a real impact.

6.1 Practice.

6.2 Go beyond rote: explain, apply, analyze, and synthesize.

6.3 Build metacognition.

As you are learning, every few hours or so, ask yourself a few questions:

What should I be paying attention to?
How well do I understand this? How difficult is this for me right now?
Do I have a strategy? Is my strategy successful?

Ask these sorts of questions often. These questions improve your metacognitive ability. And lead you to stronger learning results.

6.4 Space your practice.

The stronger the memory, the longer we keep the memory. And the more spaced repetitions of the event, the stronger the memory. Spread out your practice rather than cramming. As the memory becomes stronger, you can go longer times without review. Create a system for yourself, some sort of reminder, to review your knowledge as you learn.

Idea 7: Make It Real

We need to know not only why the knowledge is important to the subject. We also need to know why the knowledge is important to us. Real-life connection and examples gives the knowledge purpose. And makes it easier to apply the knowledge later.

7.1 Check your own reasons for learning.

Before you learn, write down why you want to learn the subject. Focus on what that goal means for you. Does getting that promotion mean more autonomy in your day-to-day work? Is that course going to help you help others in some way? Go for deeper purposes. As you learn, remind yourself of that goal every now and again. And make sure you connect the day’s goal with the bigger picture.

7.2 Connect material with real-life problems.

As you learn, try to connect what you are learning to something real to you. An easy way to do this is to ask, “What problem does this solve?” The problem is the context.

7.3 Learn from examples. Use audio, images, video, and interactive.

Listening to lectures and reading textbooks is common. But an example is so much clearer. Examples can be in visual, auditory, or interactive form. Include examples in a variety of formats in your learning plan. If you get stuck, search for an example using an image or video search engine. Or create your own diagram to help with your own understanding!

Idea 8: Learn Together

When we learn with other people, we can accelerate both our own learning and others’ learning. We can share ideas. Challenge each other’s misunderstandings. And provide and gain feedback. Learning together changes what we learn, when we learn, and how often we learn. We need to be careful in how we interact as we learn together to keep things positive and constructive.

8.1 Invite and offer feedback.

Feedback is powerful. Feedback isn’t automatic. We shouldn’t expect others to provide feedback. We need to ask for it. Feedback should be timely. Feedback should be as tailored as possible. When you are learning, see if you can find someone who can offer you feedback as you learn. And you can learn as much by offering feedback to others too.

8.2 Be honest and forthcoming.

8.3 Hold each other accountable.

8.4 Build consensus.

We are social creatures, and we are social learners. When learning with others, consensus can drive learning results. The sense of belonging and alignment can help us to learn faster.

Wrap Up

Think about ways to create an environment where these ideas can flourish. Print out this list and hang it on a wall or another place you’ll see it often.

Update: Wondering how to act on these ideas? Check out my “Checklist” article!

Thanks for reading! Feedback on this article is welcome. Let me know by writing a response! Check out the full article for more resources too.

If you’d like to know more about Sagefy, visit today.

Interviewing Prospective Users

Kevin Heis — Sat, 19 Aug 2017 00:00:00 +0000

Originally published on IndieHackers.

Maybe you’re thinking about starting a new project. Or you have a project that’s not getting results. Or you’d like to improve an existing project. Interviewing users is the best place to start! With a few quick and easy small steps, you’ll get all the information you need to get started on your next adventure.

So what are user interviews?

You’d interview 3 to 8 people for approximately 15 to 30 minutes each. Your goal is to discover who these people are, and what they need. You’ll find out their goals, the circumstances they face, and their motivations for their goals. Most important, you’ll discover their pain points. User interviews are the best way to define problems that your prospective users face.

Who am I?

I’m a software engineer in Portland, OR. I am also a designer and a user experience researcher. In addition to conducting research for my employers over the years, I’ve run several projects on the side. My most successful so far was a popular card sorting site. My current project, Sagefy, is an open-content adaptive learning platform. I’ve used user interviews extensively to help guide my projects. Even with Sagefy, I wish I had interviewed users sooner. I’m not an expert on the subject, but I will offer practical advice based on what I’ve learned about interviewing users. I’ve interviewed both real and prospective users.

Why conduct user interviews?

I have five great reasons.

The hardest part of keeping a project going is (1) motivation. Hearing from real people will give you a reason to keep working on your project. When I was in a slump on building Sagefy, I knew I had a very real problem with my tools that allow users to create content. Someone wanting to create content offered me practical feedback. That was enough for me to get back into gear and working on the project again.

The number one reason why businesses tend to fail is a lack of (2) product-market fit. The most common issue is: no one wants to use what you’ve built. By interviewing users, you start off your project already knowing your target.

User interviews can help you get started on the right foot, or they can help to correct your current path. When I was researching for Sagefy in the beginning, I conducted some user interviews. These interviews helped me find that most people faced similar issues with online learning: the content was too hard or too easy. And after I finished building my “minimally-viable product” for Sagefy, there was almost no response. After asking a few people to try the site, I interviewed them. It became immediately clear that I need to focus on developing content over developing tools.

Another great reason to interview users is (3) low cost and high reward. With less than five hours of work, and no money involved, you can get plenty of information to get the project started. It’s a low investment if you end up realizing this isn’t the project you want to take. The worst thing to do is work on something for months, possibly years, only to realize you aren’t building the right thing. User interviews can help you figure out what path to take right from the beginning. And if you’ve been on the fence about an idea, less than five hours of your time can get you moving.

A common problem founders have at the beginning of the project is everything feels abstract and overwhelming. User interviews are a great way to get more (4) concrete. Your users will become actual humans to you. Their goals and pain points will become real to you. And you’ll know what to focus on and what tasks to prioritize. When I was working on ConceptCodify, I would get many support tickets a day. I interviewed a few of the people who wrote in, and that process made my users and their needs much more real to me.

My final major reason to interview users is it will help you start and build a (5) network. You will need people as you build your project to give you feedback. And you will need people in the industry to help promote your project. It doesn’t matter how small or how big you get, you will need feedback. In fact, many large corporations have teams of employees devoted to interviewing and researching users.

How do I find people to interview?

First, you’ll need to figure out who you’d like to interview. You want to find people as close to your prospective audience as possible. Try to find different perspectives on the same process. For Sagefy, I’ve targeted post-secondary teachers, instructional designers for corporate training, and independent adult learners who have taken courses on other sites.

For finding people to interview, there are many options. You can ask friends and family members if they know anyone in your target audience. You can look up people on the web or through social media, and find their email address. You can also try going to various local events. You’d be surprised at how willing people are to be interviewed. I would guess at least 80% of people say yes.

You should try to interview about 3 to 8 people. You’ll start noticing trends around the third interview, and by the time you hit 6 or 7 people you’ll have more information than you can process. Make sure you recruit more people than you need. I would recommend trying to get 10 people to say yes and on the schedule, because a few will flake out.

When you ask people if they are open to an interview, make sure you tell them the purpose of the interview. Make it clear that the interview will not be more than 30 minutes. You can do the interview in person or over the phone; for most people over the phone is easier. If the person agrees to an interview, be the first one to suggest a specific time and date. If you leave it open-ended, that will make it harder to get the interview on the schedule.

An email thread asking someone for an interview may look like:

Me: Hello, I am Kevin and I’m thinking about building a learning platform. I’m interviewing people who have taken online learning courses to find out their goals and needs. Would you be open to a 30-minute phone call? Thank you!

Response: Yes, I’d be open to it. I’m on vacation most of next week, so this week is best.

Me: Thanks so much for your help. Would 3pm on Friday this week work for you? Feel free to suggest another time.

Now that you’ve gotten about 10 or so people on the schedule, you’ll want to prepare the questions.

What questions should I ask?

You’ll want to prepare some questions in advance. If it helps, create a checklist. Especially if you are new to interviewing users, your nerves will be higher and having a list will make the interview smoother. You’ll want to limit your number of questions. 5 to 10 is plenty.

Each interview will follow the three stage process of (1) introduction , (2) main questions , and (3) wrap up.

Don’t skip the introduction. You want to open the person up before getting into your set of questions. Tell the person who you are and what you want to get out of the interview. One time I skipped this part. Part of the way through the interview I had to backtrack. The interviewee didn’t understand the purpose of the interview, and their answers were off-track. After we went back through the introduction steps, the questions were much easier to answer.

If you want to do an audio or video recording, make sure to get permission first. I’ve never felt the need to record, but some people do.

Assure the interviewee confidentiality. Let them know you won’t go blab to their employer or colleagues about what they say. Pain points can be rather political for many organizations.

As part of the introduction, make sure you know who you’re interviewing. Ask the person about their background, role at their employer, as well as their technical familiarity. Asking these questions before you get to the entree will help you when you want to ask follow-up questions.

The questions you pick for the main section will determine what you get out of the interview.

You need to be realistic about what you can and cannot get out of a user interview. User interviews are great for figuring out who the person is, what their goals are, and their pain points. User interviews are not a good way to get specific implementation details. Don’t ask them what copy to use, which colors to use, how to lay out your navigation, or which specific features you should build. Focus on identifying problems instead of trying to come up with solutions. You are there to learn and listen; not to inform and solve. There’s other ways like usability testing on wireframes to get more specifics later.

You’ll want to focus on open-ended questions rather than closed form questions. If the person can answer it yes/no, then the question is not open-ended. So instead of, “would you use a learning platform that adapts to your prior knowledge?” you want to ask “when you are looking at a potential learning experience, what do you look for?” Closed-form answers will not get you the result you’re looking for.

Another huge call out is the difference between opinions and behaviors. People will love to share their opinion with you. However in practice, you’ll find that often the opinion doesn’t match the behavior. Your questions should focus on what people do, and not how they feel. For ConceptCodify, I had some problems on my sign up form. People would tell me that it was no problem, that they would understand what I had. But in reality, it was difficult to use because when using the site, the person is not actively thinking, but doing.

There’s a few pieces of information you want to focus on: goals, motivations, values, expectations, pain points, circumstances, resources, and attitudes.

For goals, the basic question is “what do you need to get done?” The goal is self-evident, but it’s important to establish the user’s goal before asking other questions. If you jump to pain points without knowing the goal, you’ll miss a huge piece of context. Another way to phrase it is, “what are the main things you need to get done each day?”

Once you’ve established this person’s goals, the next piece is to understand their motivations and values. The question here is “why do you need to get it done?” When you ask “why,” the person will also expose their values around the goal. Sometimes the motivation will seem obvious to you. But when you ask the question, you get a different perspective than you might initially expect.

To clarify the user’s expectations, you can ask the question “how do you currently get it done?” Usability is almost completely based on familiarity. People expect things to work the way things currently work. So how the person currently completes their goals is very likely how they expect to complete their goals in the future. Anything different from their expectations is going to be a more difficult sale. The further you go from what the person already knows, the more you have to work. By asking for the current state of things, you’ll understand what the person expects. Sometimes you need to read between the lines; not all of their expectations are going to be clear to themselves. We forget our expectations despite holding on to them anyways.

The next question is probably the one you initially think to ask: “What’s wrong with the current way you get it done?” This question reveals the person’s pain points. It is tempting to skip right to this question. But without understanding the person’s goals, motivations, and expectations first, you won’t have the full context. The previous questions help frame the interviewee’s mind as well. This is the most important part of the interview, so pay full attention and listen.

Those questions might be plenty for one interview. If so, skip to the wrap up. If time and energy allows, you might want to ask some follow up questions. A few good topics are circumstances, resources, and attitudes. These questions are in the form “How much…?” and “How often…?”. Clarify the situation the person is in when they work towards the goal. Ask them about how much time, money, people, and effort they expect to use to complete the goal. Ask them about the attitude people tend to have towards the goal and the current way of doing things.

I’ll give an example set of questions. For Sagefy, I was starting to work on building new content creation tools. I asked these sorts of questions to some instructional designers:

Goals: “When you’re creating learning content, what are the main things you are trying to accomplish?” “What types of content do you usually build?” “On a typical day, what sort of activities do you do to build content?”
Motivations and Values: “Why are these the things you try to accomplish when building learning content?” “Why do you spend so much time on that activity?” “Why do you focus on these types of content?”
Expectations: “Walk me through your current content creation process.” “What tools do you use to build learning content?” “What are the steps you follow?”
Pain Points: “What do you wish were different about your current process and tools for creating learning content?” “What do you wish other instructional designers did as part of creating learning content?” “What could be better about your current workflow?”

At the end of the interview, you should ask the interviewee if they’d like to ask you any questions. Some of the more common questions I get at the end of the interview are “what’s the next step you’re planning to take?” and “what can I do to help you?” My common answer is I’ll follow up with them after I finish the interviews and organize my thoughts. Which I would recommend doing as well!

And of course, always say thank you to the person. They’ve given up some of their time to help you. “Thank you” is a simple way to keep the connection going with the person.

Tips

It takes practice to get good at interviewing people — don’t be too hard on yourself at first. (I’m still getting better at it too!)
Focus on listening — don’t interrupt or try to change their mind. Don’t try to ‘teach’ who you’re interviewing.
Ignore solutioning — look at the underlying problems. Don’t ask about potential solutions, you’re there to define problems.
Demographics are largely worthless — look at the circumstances. What situation is the person dealing with when they are trying to accomplish their goal?
People don’t always remember everything out of context. You need to have reasonable expectations about what people are going to remember. And you can also tell them they can follow up if they remember anything else.
Facilitate — keep the person on track by using questions to come back to the focus, what you want to get out of the interview. When a person started to get too theoretical about learning, I asked a question about the task at hand to bring them back.
Be casual, don’t be formal. I had a few nervous interviews before… practice makes it easier to be casual.
Say please and thank you.
Take notes after the interview, not during. I find note taking to be very distracting when I’m trying to listen. As long as you take your notes right after, you won’t forget anything important. The larger point is you are looking for patterns in responses, and not just one person’s take.
Follow up with the person a few days later, thanking them for the interview. The person will be more likely to help you out later if you do.

Wrap Up

Don’t try to get it perfect your first try. The most important thing is just to get out there and start talking to people in your prospective audience. Everything in this article is just advice based on my own experience. You’ll find things that work better for you as you go. Stop ‘thinking about’ or ‘planning’ your project. Stop worrying about how you could theoretically make things better. Get out there and just talk to some people. Interviewing users is the best way to get started right.

If you’d like to know more about Sagefy, visit today.

What do you want to learn?

Kevin Heis — Sat, 29 Jul 2017 00:00:00 +0000

The Suggest page on Sagefy allows you to suggest new free online learning subjects. And you can up-vote subjects that interest you. And you don’t need to sign up for an account! Why use the Suggest page? Maybe you can’t find something you’re looking for. Maybe you’re thinking about building something but want to see what the interest level is first. Or you’re looking for a different approach to something that already exists.

In this article, I’m going to go into the background about the Suggest page.

What is Sagefy

Sagefy is an open-content adaptive learning system. Open-content means anyone can create and update learning content, like Wikipedia. Adaptive learning means the content changes based on the learner’s goal and their prior knowledge. The combination means anyone can learn almost anything, regardless of their prior knowledge. To learn more about Sagefy, check out this in-depth article.

How the Suggest page idea came about

So far, Sagefy is at MVP quality code-wise and I have a full electronic music intro course built out in Sagefy.

I’d like to build more content along with collaborators. I also want to continue to improve functionality. To support these efforts, Sagefy is going to need a revenue stream. Even if Sagefy does become non-profit, donations and grants are tough to come by. Finding a revenue model is a priority.

I’ve committed to myself to not put up paywalls to access learning. I’m looking for a revenue model that would also encourage content creation as well as learning. Finally, more than one revenue resources is better, to reduce risk.

I’ve brainstormed about a dozen ideas that met this criteria, but I have no idea which ones might actually work. Everyone thinks they know what product-market fit is going to be in advance, but no one does in practice. Product-market fit always seems obvious in hindsight. When you’re in the driver’s seat, the road is much more foggy. I’m not investing a ton of time in predicting and planning the perfect solution and build it all the way. I’m trying little experiments. Get out the MVP to see if the idea will work and see what the response is.

I picked the Suggest page experiment first because its had some of the lowest technical requirements of the ideas I’ve had so far. And the Suggest page meets the criteria I mentioned above — doesn’t block access to learning, encourages content creation, and encourages learning.

The Suggest page may not even seem like a potential revenue source. Right now, it isn’t creating revenue. If the Suggest page succeeds, the idea is to allow anyone to make a bid on the suggested courses, and let others pledge funds towards the bid. If the amount reaches the bid’s minimum, the content gets funded. Sagefy would take some small percentage of the funding to support Sagefy itself. I don’t need the bids or pledges to see if the idea will work. I need to know if people are willing to suggest and up-vote ideas for free online learning content in the first place. There’s many sites that already do crowdfunding of course, but I’m not aware of any strictly related to funding free online learning content. And of course, the Suggest page is one idea.

How I built the page quickly

Because the Suggest page is an experiment, I did not build the page into the main Sagefy ecosystem. The Suggest page exists as its own Nginx route. In fact, all the code is in one file. There’s not even a framework; the only library is for the database. And of course there’s no automated tests. I also did not write any JavaScript for the page. The whole thing is plain old HTML and barely any CSS. It’s about as bare bones as possible. You can build a surprising amount with this sort of setup, and get the experiment done very quickly.

Not doing things the “right” or “modern” way, I built the whole page in less than 4 hours total. I may have been able to find something out there that produces similar functionality. But I would have spent as much time searching as I did coding. With experiments going forward, I may or may not end up writing code for them… depends on the idea and what’s already out there.

If I built this page the “correct” way , this experiment would have taken 20 or more hours. Tests, frameworks, scaling, splitting off the API from the UI, and so on. Coding things the “old” way means I can do more experiments with more frequency. Its sort of fun to relive the glory days when feature expectations on the web were so much lower. If the Suggest page is successful, I’d want to update the approach to be more more feature-capable than plain old HTML.

The process for creating new experiments like this

I still have several more ideas I’d like to try out to see if I can find a revenue model for Sagefy. If you’re going through this too, here’s what the step-by-step process looks like:

Create, review, and update the criteria for revenue models.
Brainstorm ideas that would fit the criteria. Think about the user’s circumstances and what could help solve the problems they have. Also, try looking an existing models.
Pick the idea you think will meet the criteria with the least amount of effort to try the idea. Don’t be afraid of using Google Forms or other existing services to get the idea out as quickly as possible. It’s an experiment, not a commitment.
Build the experiment quickly and cheaply. Verify the idea, not necessarily actually generating revenue right away. Avoid making people do anything other than the task at hand, such as signing up for an account. Anything more than a few seconds worth of effort will deter people.
Don’t show a blank page. Add some real content before you start to promote the experiment.
Promote the experiment. No one’s going to show up because you built it. Go to sites where the audience is already browsing, and share a link.

Going forward

So far the Suggest page got a bit of initial traffic, but the interest has waned since then. I’d guess about 100 people have either suggested an idea or up-voted one. I’m not ready to count this as an unsuccessful quite yet. With some promotion the Suggest page could have potential. I don’t have a ‘exit plan’ as there’s no costs involved, so the Suggest page is going to be around for the foreseeable future.

I’m going to try some more experiments too. All while also continuing to improve Sagefy’s learning and contributing experiences. So… keep your eyes open for improvements and more experiments too!

Try out the Suggest page

If you’ve made this far in the article, congrats! That, and you should try out the Suggest page. No need to sign up for an account. You can up-vote existing ideas and add your own. And let me know your feedback too!

If you’d like to know more about Sagefy, visit today.

The Mathematics of Sagefy

Kevin Heis — Thu, 06 Apr 2017 00:00:00 +0000

In this article, I’m going to talk about some of the mathematical models I’m using in Sagefy so far. I’m going to discuss some of the problems I’ve run into, and what problems are still ahead. I’m hoping to occasionally update this article as the math that Sagefy uses changes.

What is Sagefy?

Sagefy is an open-content adaptive learning system. Open-content means anyone can create and update learning content, like Wikipedia. Adaptive learning means the content changes based on the learner’s goal and their prior knowledge. The combination means anyone can learn just about anything, regardless of their prior knowledge. To learn more about Sagefy, check out this in-depth article.

The Sagefy Learner Experience

After you create an account, the first thing to do is to find a subject. A subject is like a course or a collection of courses. You can think of it like folders on your computer; they nest. You add the subject to your list of subjects. Then, when you start the subject, you’ll see a tree of units . Each unit is a single learning objective. Then, Sagefy will ask you to choose which unit you’d like to start with. Sagefy presents a recommended unit based on what you know. After choosing a unit, Sagefy will show you a card. A card is a single learning activity, like a video or multiple choice question. As you use the cards, Sagefy updates it’s model of how much you know the unit. When you’ve learned the unit, you choose another. The trio of subject, unit, and card define the learning experience.

Popular Models of Learning

There’s a few different learning models out there. The subject is diverse. “Building Intelligent Interactive Tutors” by Beverly Park Woolf is a more comprehensive treatment. Here’s some of the major ones:

Forgetting Curve

The Ebbinghaus Forgetting Curve is one of the oldest models. Depending on the strength of the memory, our ability to remember generally follows this formula:

In this formula, r is retention, tau is time, and s is the strength of the memory.

Learning Curve

The opposite of the forgetting curve is the learning curve:

Where a is learning. Some authors suggest an alternative model for learning, a sigmoid function:

Bayesian Knowledge Tracing

Bayesian Knowledge Tracing determines how likely a learner knows a skill, based on the learner’s responses.

Bayesian Knowledge Tracing derives from Bayes’ Theorem.

| |

We call…

p(A|B) the posterior — what we believe after seeing the data.
p(A) the prior — what we believe before the data.
p(B|A) the likelihood —given our prior belief, how likely was the data.
p(B) the normalizer — how likely is the data given all hypotheses.

p(B) is often difficult to formulate. We can use this as a replacement:

| |

~ in this case means not.

For BKT, we have the follow factors:

p(L) — probability the learner learned the skill.
p(C) — probability the learner will get the item correct.
p(T) — probability the learner will learn the skill on a particular item.
p(G) — probability the learner will guess the right answer.
p(S) — probability the learner will mess up even knowing the skill.

For any item, the probability of getting the answer correct is:

Putting this all together, the probability the learner has learned the skill is, given a correct answer:

Given an incorrect answer:

We also must account for the idea the learner may have improved the skill during the card. So…

The power of Bayes is the ability to update the model each time we receive new information. We don’t have to operate on larger data sets at less frequent intervals. We can constantly update the model just a little bit each time.

For a better explanation, I recommend this video from Ryan Baker’s Big Data and Education course.

Item Response Theory

Item Response Theory determines how likely a learner will correctly answer a particular question. It is a logistic function. We use IRT is more commonly in testing, but we can use IRT in adaptive learning as well. The parameters read:

theta — learner ability.
b — item difficulty.
a — item discrimination; how likely the item determines ability.
c — item guess.

There are two common formulations:

The formulas change slightly depending on author.

We can extend Item Response Theory into Performance Factors Analysis. PFA is a competing model with Bayesian Knowledge Tracing.

Knowledge Space Theory

Knowledge Space Theory represents what skills learner knows. KST derives from antimatroids.

We assume a learner has either learned a skill or not. Given skills +, -, *, and /, we would form prerequisites, such as:

+ -> -
+ -> *
* -> /

The knowledge space represents all possible sets of knowledge a learner might have, such as:

none
+
+, -
+, *
+, *, /
+, -, *, /

An individual learner has a likelihood for each of these sets. KST makes the assumption that an individual question may inquire about multiple skills. We begin by asking questions that use multiple skills, and work backwards to assess learner knowledge.

Several automated systems exist to automatically determine prerequisites based on learner performance.

Spaced Repetition

Spaced Repetition suggests that learners should spread out their practice. Reviews should happen less frequently as ability improves.

The most popular algorithm is SuperMemo 2. The first review is after 1 day, the second review is after 6 days. After which, the next review is:

…where e is how difficult or easy the item is. e is between 1.3 and 2.5, and it uses learner responses on a Likert scale to determine the next time to review.

Later versions of SuperMemo include other considerations, such as:

Similar cards
Previous iteration duration
Ebbinghaus forgetting curve

The latest is version 11/15.

Why Sagefy uses Bayesian Knowledge Tracing

Sagefy uses Bayesian Knowledge Tracing. The reason is that it is a simple formula and computationally inexpensive. The other advantage is that with Bayes, the model updates each time the learner interacts with the system a little bit each time. This feature avoids expensive modeling computations all at once. That means that Sagefy can both scale more easily and handle a larger diversity of content.

Adding Time to Bayesian Knowledge Tracing

Bayesian Knowledge Tracing does not consider forgetting. To account for forgetting, I have added a taste of the Ebbinghaus forgetting curve into the final calculation. A new formula which calculates belief, which is our confidence in p(L) given the elapsed time:

exp(-1 * time_delta * (1 — learned) / belief_factor)

I set my belief factor to 708000 based on guess-and-check. Likely, I could optimize this value later on.

How I’m Working to Learn Card, Unit, and Subject Parameters

Everything here so far lays out how to estimate how likely the learner knows the unit. But, what the other parameters, and how do we come to know those? This is the more challenging area.

Parameters of Cards, Units, and Subjects

Cards. In Bayesian Knowledge Tracing, most implementations assign a Guess , Slip , and Transit at the unit level. But, for Sagefy, to get deeper information, I’ve assigned those parameters to cards. Transit informs us of the quality of the card. Guess and Slip informs us of the difficulty of the card. Difficulty helps us to choose cards appropriate to the learner’s ability. We also want to track the number of learners who will we will impact when we make changes to the card.

Units. We want to know a way of determining approximately how much time or how many cards it takes on average to learn the unit. This is the difficulty of the unit. We can judge the quality of the unit based on the average ability of the learners of the unit. We also want to track the number of learners who will we will impact when we make changes to the unit.

Subjects. We want to know a way of informing the learner how much time it will take to complete the subject, given their currency state of progress. This is the difficulty of the subject. We can calculate the difficulty of the subject by summing up the difficulty of the remaining units. We can also calculate quality in a similar fashion, but using a mean instead of a sum. We also want to track the number of learners who will we will impact when we make changes to the subject.

You can see we have lots of data about the smallest entities, and less data about higher entities. We track fine data about cards, a little less about units, and subjects just aggregate unit data. So for this section of the article, the focus will be on card parameters.

Probability Mass Functions

Bayes gives us the ability to make constant updates to our models instead of making big and heavy calculations all at once. To achieve this, we work with distributions. So instead of just resulting with a number, we have a line chart. On one axis is the possible values, on the other axis is the probability of that value being the correct answer. So as we get more data, the graph tightens around a single value.

There’s two types of distributions. One is continuous. In a continuous distribution, we have a single equation that represents the entire graph. The other is discrete. So we have specific data points that represent our graph. Computers naturally can work with discrete much more quickly than continuous. The official name of a discrete distribution is the probability mass function. Because Sagefy is ultimately made of servers, the PMF is the more efficient and easier to use choice.

PMFs are easy to work with in Bayes. Each possible value gets its own update that matches the p(A|B) = p(A) * p(B|A) / p(B) formula. Then we normalize the PMF after the update so that the sum probability of the values is 1. With the PMF, the normalizer takes care of p(B). And we already have p(A) which is the previous set of hypotheses. That means we need a way to compute p(B|A), the likelihood.

Everything I know about probability mass functions I learned from the book Think Bayes by Allen Downey. He gives a much better explanation than I could, and the book is free.

Card Parameters

We need to update Guess, Slip, and Transit per learner response. Guess should increase with correct answers and decrease with incorrect answers. Slip should increase with incorrect answers and decrease with correct answers. Transit should increase if the later responses are more correct, and decrease if the later responses are less correct.

For Guess and Slip, I figured how how to calculate the likelihood, though it doesn’t work as accurately as I’d like. The formula for if a learner will get an answer correct is:

learned * (1-slip) + (1-learned) * guess

and incorrect…

learned * slip + (1-learned) * (1-guess)

If the learner got it right, I can use the first formula as the likelihood. If the learner got it wrong, I can use the second formula. Learned is the learner’s learned. You take the prior slip to calculate guess, and the prior guess to calculate slip. The value of guess or slip is just each hypothesis (so you do this calculation per each hypothetical value). Doing the math, it means earlier learners will impact guess more, and later learners will impact slip more as expected.

When I ran this in simulation, it held a strong correlation to the initial values. But, they were off by scale. I have manually adjusted the scale in Sagefy. Transit is likely at play at throwing off these calculations, but I was unable to find a formula to take out this factor automatically. This is an opportunity for improvement.

I was unable to find a way to calculate a likelihood for transit. I know the following is true:

transit = (learned_post — learned_pre) / (1 — learned_pre)

This likelihood never panned out in simulation, so for now Sagefy has a set transit for all cards. This is a major opportunity for improvement.

I haven’t figured out how how to calculate the number of learners quite yet. That’s more of a technical challenge than a mathematical one.

Unit and Subject Parameters

I haven’t done the work to calculate the two unit and two subject parameters: difficulty and number of learners. These are both essentially database queries. The responses table can inform the difficulty. Learners subscribe to subjects, and not units. The solution requires figuring out all the relationships and then counting from there. Not impossible, but not easy either.

How I’m Simulating Models and Error Rates

Because Sagefy is new, I don’t have much real user data to test my model against. For now, I’m using simulated data. I created a script that will create a random collection of subjects, units, cards, and users, all of each with random parameters within ranges. I’m using triangle ranges instead of completely linear randomization to more closely simulate real data. Then, I can run my model against the simulated data, and take error rates. We can calculate the error rate for each parameter as:

Where s_i is the simulated value (each example), r_i is the real value, n is the number of examples.

When I modeled this, I was able to see my errors shrink by changing the number of steps in my PMF. I also saw improvements by scaling down guess and slip… again, that’s still a mystery at this point. Hopefully, if Sagefy attracts users, we will be able to create better models using real data instead of simulated data.

Traversing the Graph of Units

Bayesian Knowledge Tracing helps us within a unit. But, subjects contain many units. To work with the course, we must iterate over the graph of units.

We collect the subject of units that the learner will be participating in. We will need to diagnose any units which the learner hasn’t seen. We will also need to diagnose any units that the learner viewed, but we are no longer confident in the ability score due to time.

The following is an example of this process, known as a graph traversal.

The algorithm makes use of depth first search. We start near the end of the tree, and walk our way down as we diagnose. We record each node in one of three lists: Diagnose, Ready, and Learned.

First, we start at node “A”. We diagnose an L (low score). Because it is a low score, we will continue to traverse the tree. Append “A” to list Diagnose.

Next, we will continue to node “C”. This is because we have higher confidence in “C” than “B”. “C” is already diagnosed. Append “C” to list Ready. We note that “C” has one dependency, “A”.

Third, we continue to node “F” (depth-first search). We are confident that the learner knows node “F”, and we start on the other side of the chain. Append “F” to Learned. Although “F” requires “I”, because we are confident in high ability for “F”, we will not diagnose “I”.

Because “C” has no other requires, we go back to “B”. We find it is a low ability. Append to Ready, 1 dependent: A.

We know more about “E” than “D”, so we continue to “E”. We find “E” is a low ability. Append to Ready, 2 dependents: A and B.

We already know “G”, so append “G” to Learned.

We diagnose “H”, and find it is low ability. Append to Ready, 3 dependents: “A”, “B”, and “E”. If the learner hadn’t learned “F”, unit “H” would have 4 dependencies instead of 3. The algorithm considers how many nodes depend on the given node, rather than how deep in the graph the node is.

Finally, we diagnose node “D” with low ability. Append to Ready, and there are 2 dependents: “A” and “B”.

We also have the following lists:

Diagnose: [] Ready: [A, C, B, E, H, D] Learned: [F, G]

We are now ready to starting the learning process.

The following is overly simplistic; most learners will not ‘learn’ a unit in their first attempt at it. If at any time the unit composition changes, when we come back to the tree, we will need to diagnose any new units. Additionally, learners will need to review some units every now-and-again to keep the confidence scores up.

In the ready list, A, B, and E have requires, so those are not options to the learner yet. The available ready nodes are C, D, and H.

“H” has the most dependents, so Sagefy would recommend “H” as the starting place for the learner. Let’s say, though, the learner choose to learn unit “D” first.

Now the remaining set is C and H. The learner chooses H. So now the remaining set is E and C. Let’s say the learner chooses C.

Now, the only option remaining is E. After E, the learner would do B, then A.

Choosing Cards

After the learner selects a unit, then Sagefy must choose a card for them. Several factors come into play:

How well has the learner mastered this unit? p(L)
Which cards are assessment v non-assessment cards? (e.g. Videos are non-assessment, multiple choice is assessment)
Has the learner recently seen the card?
Does the card require another card first?
How good is the card? p(T)

Our goals are as follows:

We want to show cards of appropriate difficulty. A card with high slip would not be appropriate for someone with a low p(L), for example.
We want to focus on non-assessment cards for low p(L), and assessment cards for a high p(L).
We want to avoid showing a card the learner has seen recently, when possible.
We want to ‘follow’ card requires.

The current process works like this:

Find p(L).
Choose ten random cards in the unit.
Split up the cards by assessment versus non-assessment.
If the learner has a low p(L), be more likely to prefer non-assessment. As p(L) increases, prefer assessment.
For an assessment card, try to find one where the probability of correct is greater than 0.25 and less than 0.75. Otherwise, show any assessment card.
For non-assessment, just pick the first one.
Otherwise, choose any remaining card available.

A few things could improve here. We can add tags, track how the learner does with different kinds of cards, and use p(T) to choose better quality cards. Overall, this is not as difficult of a problem as figuring out what the learner does and does not know.

Contributor Experience

The contributor experience is still a work in progress.

In most open-content systems, anyone can edit. But, usually there’s a number of people subscribed to all changes to the content. When a change occurs that was not discussed, the change is immediately reverted. Only by discussion and consent is the change made permanent.

Sagefy codifies this process by not allowing direct edits, but by allowing anyone to propose changes to content. Then, other users can vote on the proposal. With enough approving votes, Sagefy makes the change. Sagefy keeps a record of all the changes too. With any significant dissent, Sagefy blocks the proposal.

Contributor Ratings and Consensus-Based Decision Making

The decision to accept a proposal has a few factors: the reputation of the proposer and voters and the usage of the entity.

Entity Calculation

We calculate the “Yes” vote power as the sum of the proposer and the “yes” voters’ vote power.

We calculate the “No” vote power as the sum of the “no” voters’ vote power.

To accept a proposal, the proposal requires log2(number_of_learners) in “yes” vote power.

To block a proposal, the proposal requires log100(number_of_learners) in “no” vote power.

To unblock a proposal, we must work together to reduce the “no” vote power to below this amount. Otherwise, the proposer can decline their proposal.

The reason for this is simple: its likely impossible to meet perfect consensus all the time. Yet, we should strongly consider dissent. Majority rule isn’t as powerful as near-consensus.

Example:

Number of Learners, To Accept, To Block - 1, 0, 0 - 10, 3.32, 0.5 - 100, 6.64, 1 - 1000, 9.97, 1.5 - 10000, 13.28, 2 - 100000, 16.61, 2.5

Contributor Calculation

Each proposal and vote for a proposal in accepted state counts as +1 point. Sagefy calculates the vote power as 1-e^(-points/40). This may seem like a convoluted formula. The reason is to avoid “superusers” who by their linear reputation alone can make any change. The difference between 0 and 10 successful contributions gives us lots of information. 10 to 100, less so. And 100 to 1000, even less. We learn the most from the earliest examples, and less with each example.

Example:

Points: Power - 0: 0 - 5: 0.12 - 10: 0.22 - 20: 0.39 - 50: 0.71 - 100: 0.91 - 150: 0.97 - 200: 0.99

Call to Action

As you can see, this is just the beginning. Sagefy is right now the seed of what it will grow into. To review, here’s the most immediate open questions and TODO items:

Why does the likelihood function overestimate guess and slip?
How do we compute the likelihood for transit?
Calculate difficulty and number of learners for units and subjects.
Efficiently diagnose the learner’s current knowledge before entering ‘learn mode’.
Test using real data instead of simulated data.
Implement contributor rating and consensus system.

Some of the other opportunities to use math in Sagefy:

Improve the model for choosing cards.
Create a model for handling asynchronous peer assessment. (Such as short answer/essay type questions, file uploads, etc.)
Recommend new subjects to the learner.
Filter spam and low quality content.
Match up learners with other learners and mentors.
Group learning.
When should the learner take a break?
How valid or accurate is the content?

If you find this interesting, come join in! Reach out on Twitter . Thank you!

If you’d like to know more about Sagefy, visit today.

Gamification Isn’t Always the Right Answer

Kevin Heis — Sat, 01 Apr 2017 00:00:00 +0000

Gamification seemed like a natural fit for Sagefy. The term had been around since about 2010. Gamification came to its current position in 2013. 2013 was around the same time I started working on Sagefy. I didn’t know much about gamification beyond the cliché of points, badges, and leaderboards. But I did know that many businesses and organizations invested in gamification. In particular, those related to learning (Khan Academy, Duolingo). I explored this through videos, articles, books, and courses. I realized people used gamification as a synonym for a more familiar phrase: extrinsic motivation.

The word “gamification” came into existence around 2010. “Gamification” reached the term’s current standing in 2013.

What is Sagefy, and What is Gamification

Sagefy is an open-content adaptive learning system. Open-content means anyone can create and update learning content, like Wikipedia. Adaptive learning means the content changes based on the learner’s goal and their prior knowledge. The combination means anyone can learn just about anything, regardless of their prior knowledge.

Gamification is the application of game-like elements to traditionally non-game activities. An example of gamification would be awarding points to a sales team on each sale, and awarding a prize for the highest number of points. In learning contexts, this usually means something like awarding points for mastering content, and awarding prizes like trophies or unlocking content.

We often find gamification in learning contexts, especially e-learning. The reason is motivation. Online courses have typically lower participation and completion rates than traditional courses. Learning content creators use gamification to improve their statistics.

What I’ve Read

I’m no expert in gamification. Given the age of gamification, I’m not sure anyone is. That said, I’ve had a little background on it. Some of my influences include:

The Coursera Gamification course, taught by Kevin Werbach.
“We Don’t Need No Stinkin’ Badges”, by Jane McGonigal.
The Gamification of Learning and Instruction by Karl Kapp.

These are three specific items. Also, I have a background in user research and testing, as well as my interest in learning science and memory.

Pros and Cons of Gamification

Some of the good things can come from adding game elements to a learning platform like Sagefy. The largest opportunity is of course increased motivation and engagement. Gamification can lead learners to use the site for longer and to return more frequently. Adding playful elements can increase the positive emotions of the platform. Positive emotions help to spread word-of-mouth. Gamification can add a sense of progress, and can increase social activity.

There’s some cons. We already see some negative reaction to points, leaderboards, and badges. Because gamification is becoming so common, many people associate these game elements with marketing. Gamification can also be manipulative, both intentional and unintentional. By setting up the wrong motivations, it can lead learners to focus on gaming the system rather than on learning. Gamification can also lead learners to focus on the gamified rewards rather than the reward of learning. This can lead to a set of false expectations. Rewards can demotivate because they distract from the learner’s internal goals. The other nuance is that different learners have different reasons for learning. Not everyone finds all game elements motivating.

Why Intrinsic Focus for Sagefy

In my mind, adding some gamified elements can be fun. The most important thing is that those elements support, and not oppose, the internal motivations of the learner. For example, giving users an easy way to track their progress is important. Reminding users of their goals is valuable. Encouraging social interactions can add to the learning experience. Focusing too much on badges, leaderboards, economies, and points can detract instead.

Sagefy is currently minimal. But it will grow in features. The question for me with motivation will remain. Does this feature support the learner’s intrinsic motivation? Or does is distract? If it doesn’t pass that test, I can’t support it.

If you’d like to know more about Sagefy, visit today.