DEV Community: dayvonjersen

How to use io.Reader and io.Writer

dayvonjersen — Fri, 12 Apr 2019 07:21:04 +0000

This was confusing for me when I started using Go after coming from other languages.

Most of the technical articles and tutorials I came across made numerous references to io.Reader and io.Writer, interfaces provided by the io package, describing them as excellent and easy-to-use but seemed to assume that the person reading already knew how to use them.

Those abstractions are indeed powerful and convenient so here's how to actually use them in some common scenarios.

io.Reader

Say you have some reader which implements the io.Reader interface and you want to get the data out of it.

You could make a slice of bytes with a capacity, then pass the slice to Read().

b := make([]byte, 512)
reader.Read(b)

Go slices are reference types which means when Read() is called, b will be modified to contain the data from the reader.

However, Read() will only copy as many bytes as the passed slice has capacity for.

Most of the time, gophers will reach for ioutil.ReadAll when they simply want all of the data at once.

You could also use a byte buffer and io.Copy.

For example, here's a function I made and use all the time to get the contents of text files as a string (inspired or perhaps spoiled by php's file_get_contents())

func fileGetContents(filename string) string {
    contents := new(bytes.Buffer)
    f, err := os.Open(filename)
    checkErr(err)
    _, err = io.Copy(contents, f)
    if err != io.EOF {
        checkErr(err)
    }
    checkErr(f.Close())
    return contents.String()
}

checkErr() above is just if err != nil { panic(err) }

But what if reader is a stream of data with indeterminate (or neverending) length?

In that case you want to reach for a bufio.Scanner:

scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
    fmt.Println(scanner.Text())
}

io.Writer

To write data is very straightforward:

someBytes := []byte("Hello world")
f, err := os.Open("someFile.txt")
checkErr(err)
f.Write(someBytes)
f.Close()

Of course io.WriteString is less typing:

f, err := os.Open("someFile.txt")
checkErr(err)
io.WriteString(f, "Hello world")
f.Close()

A simple byte buffer

There's nothing stopping you from implementing io.Reader and io.Writer in your own user-defined types.

For example, here's a very useful, simple byte buffer:

type buf []byte

func (b *buf) Read(p []byte) (n int, err error) {
    n = copy(p, *b)
    *b = (*b)[n:]
    if n == 0 {
        return 0, io.EOF
    }
    return n, nil
}

func (b *buf) Write(p []byte) (n int, err error) {
    *b = append(*b, p...)
    return len(p), nil
}

func (b *buf) String() string { return string(*b) }

I made it at first just to capture the text output of shell commands in go.

ʕ◔ϖ◔ʔ

io.Reader and io.Writer are used all over the standard library from shell commands to networking even the http package!

Hopefully now you know the basics of how to use readers and writers and maybe you can even try to implement io.Reader and io.Writer for your own user-defined types.

C++ For Go Programmers: Part 3 - Maps

dayvonjersen — Thu, 06 Sep 2018 09:22:55 +0000

DISCLAIMER: This series title is inspired by a terrific article on the Go Wiki but, unlike that article, none of the information presented in this series should in any way be taken as sound advice. The author presumes on the part of the reader a competency in both modern C++ and Go which far exceeds the author's own. This is actually a thinly-veiled game devlog

map is easily the worst built-in datatype in Go. I have an outline of the reasons why but we're not about negativity here. Besides, I'm pretty sure someone else has already written that article.

Nope, we're not about that life. We're about exploring wonderful and exciting new possibilities in C++!!! ...by a combination of reinventing the wheel, baby duck syndrome, and eschewing well-defined best practices.

if ok {

In Go, you can access a map value and check if it exists in one step:

m := map[string]int{
    "one": 1,
    "two": 2,
}

one, ok := m["one"]     // 1, true
two, ok := m["two"]     // 2, true
three, ok := m["three"] // 0, false

std::map in C++ does have operator[] but its behavior is different:

Returns a reference to the value that is mapped to a key equivalent to key, performing an insertion if such key does not already exist.

That's not what we want.

std::map::find however, will allow us to determine if a value exists first.

We don't have multiple return values in C++ so we'll have to pass a variable by reference to get the value.

Here goes nothing:

#include <map>

template<typename K, typename V>
bool mapValue(std::map<K,V> m, K key, V& value) {
    auto it = m.find(key);
    if(it != m.end()) {
        value = it->second;
        return true;
    }
    return false;
}

Adapting the Go example above somewhat, use it like this:

std::map<std::string,int> m;
m["one"] = 1;
m["two"] = 2;

int one;
bool success = mapValue(m, std::string("one"), one);
if(success) {
    // ...
}

w00t

This is actually the first time I've made a templated function and it was actually kinda fun even though I know I'm going down the path to ruin here (first you start sprinkling your code with some nice little generics every so often then you start doing metaprogramming next thing you know you're writing compile-time tetris and contributing to Boost)

It should be noted that the implementation here won't work with all types, notably C-style char* strings (hence why I used std::string in the example usage).

And also I probably shouldn't be using std::map at all for storing information for special tiles in my tilemaps but I'm not worried about it it's fine don't worry about it shhhhh

C++ For Go Programmers: Part 2 - Slices

dayvonjersen — Wed, 15 Aug 2018 18:21:36 +0000

DISCLAIMER: This series title is inspired by a terrific article on the Go Wiki but, unlike that article, none of the information presented in this series should in any way be mistaken as good advice. See Part 0 for an introduction.

Seriously, don't do what I'm about to demo in this article. If you're using C++, just use std::vector. If you're using C, check out stretchy buffers.
EDIT: Also I got slice semantics wrong. s1 := s0[:] in Go actually makes both refer to the same underlying array, instead of making a copy. And realloc() could screw things up too. Thanks to jstag on #go-nuts for pointing these mistakes out.

As a thought experiment, let's try to implement Go slices in C and C++. If you don't want the play-by-play, take a look at the finished product in C and C++.

Part of what makes Go slices so nice is the convenient syntax:

s0 := []int{1, 2, 3}       // literals
s1 := make([]byte, 0, 512) // or specify length and capacity
s2 := s0[1:]               // s2 == {2, 3}, s0 is still == {1, 2, 3}
s3 := s0[:] // create a copy of a slice
s3[2] = 5   // modify in-place
somethingThatImplementsioReader.Read(s1) // slices are reference types, s1 has data in it now (maybe)
s0 = append(s0, 4, 5, 6) // append implicitly resizes
s1 = append([]int{0}, append(s0[:1], s0[2:]...)...) // like I said, nice!

We'll have none of that happy nonsense. This is C.

NOTE: Go also has the very nice feature of being able to effortlessly convert its native utf8 string type to and from either []byte or []rune types. I have no idea how to accomplish this in C/C++.

Slices in C

A slice has a length, a capacity, and a pointer to an underlying array.

typedef struct _slice {
    int   len;
    int   cap;
    void* arr;
} slice;

The builtin functions len and cap let you access the length and capacity of a slice.

int len(slice* s) { return s->len; }
int cap(slice* s) { return s->cap; }

So far this is pretty simple and straight-forward.

But actually, we need to store the type information. Types are not first-class values in C, so we'll store the size of the type in bytes and hope that's good enough. We'll come back to this later.

typedef struct _slice {
    size_t type;
    int    len;
    int    cap;
    void*  arr;
} slice;

The builtin function make allocates a slice with a specified length and optionally a capacity.

We'll require both and additionally the size (in bytes) of the type of the elements of the array as the first argument.

slice* make(size_t type, int len, int cap) {
    slice* s = calloc(1, sizeof(slice));
    s->arr   = calloc(cap, type);
    s->type = type;
    s->len  = len;
    s->cap  = cap;
    return s;
}

Slicing a slice creates a new slice. We can't do bracket notation, however, so we'll make it a function.

// naming stuff is hard
slice* sslice(slice* s, int start, int end) {
    int len = end-start;
    slice* s2 = make(s->type, len, len);
    memcpy(s2->arr, s->arr + start * s->type, len * s->type);
    s2->len = len;
    return s2;
}

Here's where the fun begins with the pointer arithmetic. As you know, arr[i] is just shorthand for *(arr + i), but the compiler needs to know the type of arr in order for this to work. We're working with void* here so we have to multiply the offsets by the size of the type in bytes.

"The copy built-in function copies elements from a source slice into a destination slice. [...] The source and destination may overlap. Copy returns the number of elements copied, which will be the minimum of len(src) and len(dst)."

https://godoc.org/builtin#copy

int copy(slice* dst, slice* src) {
    if(dst->type != src->type)  return 0; // should generate an error somehow actually
    int len = dst->len < src->len ? dst->len : src->len;
    memcpy(dst->arr, src->arr, len * src->type);
    return len;
}

We have enough now to start using these slices, but append() is really what makes Go slices so valuable.

If we knew the type of the underlying array we could do this:

typedef struct _int_slice {
    int  len;
    int  cap;
    int* arr;
} int_slice;

// same len, cap, make, sslice, and copy... except you could could omit type and use sizeof(int)

int_slice* append(slice* s, int elem) {
    if(s->cap < s->len+1) {
        s->arr = realloc(s->arr, sizeof(int) * s->len+1);
        s->cap = s->len+1;
    }
    s->arr[s->len++] = elem;
    return s;
}

But since we want to have slices of any type, we have to use void*.

NOTE: As a consequence, we cannot pass literals or value types to our append(), we always have to pass a pointer type. Something like int a = 5; s = append(s, &a); worked in my little toy examples but it's not great.

Recall that in C a byte is an unsigned char, and we have the size in bytes of the type of each individual element stored, thus:

slice* append(slice* s, void* elem) {
    if(s->cap < s->len+1) {
        s->arr = realloc(s->arr, s->type * s->len+1);
        s->cap = s->len+1;
    }
    int offset = s->len * s->type;
    for(int i = 0; i < s->type; i++) {
        *((unsigned char*)s->arr + offset + i) = *((unsigned char*)elem + i);
    }
    s->len += 1;
    return s;
}

"Ah", I can almost hear you say now, "but the real append() takes variadic arguments!" I got you covered, family:

slice* append(int count, slice* s, ...) {
    va_list args;
    va_start(args, s);

    if(s->cap < s->len+count) {
        s->arr = realloc(s->arr, s->type * s->len+count);
        s->cap = s->len+count;
    }

    int offset = s->len * s->type;
    for(int j = 0; j < count; j++) {
        unsigned char* elem = va_arg(args, unsigned char*);
        for(int i = 0; i < s->type; i++) {
            *((unsigned char*)s->arr + offset + i + j*s->type) = elem[i];
        }
    }
    va_end(args);

    s->len += count;
    return s;
}

Unfortunately, we have to specify how many elements we're appending because <stdarg.h> provides no mechanism for counting the number of variadic args. This is entirely consistent with how C expects the programmer to constantly keep track of the length of arrays and will happily read off the end and Segfault on your face, leading back around to why you would want to use something like slices in the first place.

full source code: slices.c 👀

Don't forget to free() your slices and their underlying arrays when you're finished with them.

Slices in C++

So now we have Go slices in C, but C++ (and its creator) will scream at us if we try to compile this code. It's just a matter of casting the return value of *alloc and ignoring and/or turning off some compiler warnings, but let's try to do it the right way™ with classes and templates:

template<typename T>
class slice {
    int _len;
    int _cap;
    T*  _arr;
public:
    slice(int len) : slice(len,len) {}
    slice(int len, int cap) {
        _len = len;
        _cap = cap;
        _arr = (T*) calloc(cap, sizeof(T));
    }
    ~slice() {
        free(_arr);
    }

    int len() { return _len; }
    int cap() { return _cap; }

    T& operator[](int i) { return _arr[i]; }

    void append(T elem) {
        if(_cap < _len+1) {
            _arr = (T*) realloc(_arr, _len+1 * sizeof(T));
            _cap = _len+1;
        }
        _arr[_len++] = elem;
    }

    template<typename... Targs>
    void append(T elem, Targs... args) {
        append(elem);
        append(args...);
    }

    slice<T> sslice(int start, int end) {
        int len = end-start;
        slice<T> s2(0, len);
        for(int i = 0; i < len; i++) {
            s2.append(_arr[start + i]);
        }
        return s2;
    }
};

full source code: slices.cpp 👀

Templates let us have slices of any type without having to do any pointer arithmetic or obscene casting.

I think I'm starting to sound a little too much like Bjarne so I should skip the play-by-play and wrap this up quickly.

NOTE: I had to use stdlib *alloc and friends because new[] for an unbounded array class member led to some strange results...

Bonuses for using C++:

We get to have free() called in the destructor automatically
Bracket notation for accessing slice elements directly.
Constructor overloading lets us get the same arguments as Go's make() (can omit capacity)
C++11 introduced parameter pack for variadic arguments, very nicely explained here by this fine fellow. I'm not crazy about having to use recursion but realistically you'd never append enough elements in a single call to overflow the stack.

The End

In conclusion, don't do this. You get none of the convenient syntax of Go slices nor any of the flexibility of conversions between Go strings and []byte or []rune while having to manage your own memory. realloc()'ing all the time can't be good for performance and you'll leak memory if you forget to free() every slice, especially since slicing a slice creates a new slice every time. Again, if you're using C++ just use std::vector otherwise check out stretchy buffers. Both of those check for OOM conditions and weren't hacked together in 30 minutes to write a follow-up article for a series that isn't even really a series.

C++ For Go Programmers: Part 1 - Struct Embedding, Object Receivers, and Interfaces in C++

dayvonjersen — Sat, 11 Aug 2018 10:28:04 +0000

DISCLAIMER: This series title is inspired by a terrific article on the Go Wiki but, unlike that article, none of the information presented in this series should in any way be taken as sound advice. The author presumes a core competency in both modern C++ and Go on the part of the reader which exceeds the author's own. It is a thinly veiled call for help from a disturbed individual with too much time on his hands and an irrational fear of traditional OOP. See Part 0 for an introduction.

Struct Embedding

Go provides struct embedding as ~~the only~~ an alternative to inheritance:

package main

type parent struct {
  something int
}

type child struct {
  parent
  somethingElse float64
}

func main() {
  c := child{}
  c.something = 42
}

But if you fmt.Printf("%#v", c) you'd see how this actually works:

main.child{
  parent:main.parent{something:42},
  somethingElse:0
}

And indeed you can access something by doing c.parent.something as well.

Therefore, on a conceptual level at least, the C++ equivalent is:

struct parent {
  int something;
};

struct child {
  parent parent;
  double somethingElse;
};

int main() {
  child c = {0};
  c.parent.something = 42;

  return 0;
}

The same holds true for C, barring some typedef's and this is nothing new. C programmers do this sort of thing all the time.

Ergo, we can have inherited members by using struct fields.

Object Receivers

Go allows us to define methods on our defined types. Not only with structs, but let's keep things simple.

type myStruct struct {
  something int
}

func (s *myStruct) Method() {
  s.something = 42
}

The object receiver is of a pointer type, rather than as a value in order to effect changes when called

s1 := &myStruct{}
s1.Method() // s1.something == 42

Conceptually, this is equivalent to:

func Method(s* myStruct) {
  s.something = 42
}

And indeed the C/C++ equivalent is something very commonly found in C code:

struct my_struct {
  int something;
};

my_struct_method(my_struct* s) {
  s->something = 42;
}

Object receivers as the first argument to a function which acts on a struct is the defacto way of declaring "methods" on an "object" in C and is valid C++. This is nothing new.

Interfaces

Let's combine what we've established so far and do something completely bananas.

Go's interface model, to paraphrase Commander Pike, is

If an object has all the methods of an interface, then it satisfies that interface.

Have some code:

#include <stdlib.h>

#include <functional>
#include <iostream>
#include <vector>

struct some_interface {
  std::function<void()>* Method;
};

some_interface* new_some_interface() {
  return (some_interface*) calloc(1, sizeof(some_interface));
}

struct parent {
  some_interface* some_interface;
};

parent* new_parent() {
  parent* p = (parent*) calloc(1, sizeof(parent));
  p->some_interface = nullptr;
  return p;
}

struct child {
  parent* parent;
  int something;
};

child* new_child() {
  return (child*) calloc(1, sizeof(child));
}

child_method(child* c) {
  c->something = 42;
}

int main() {
  child* c1  = new_child();
  child* c2  = new_child();
  c1->parent = new_parent();
  c2->parent = new_parent();

  c1->parent->some_interface = new_some_interface();
  c1->parent->some_interface->Method = new std::function<void()>([=]() -> void {
    child_method(c1);
  });

  std::vector<parent*> parents;
  parents.push_back(c1->parent);
  parents.push_back(c2->parent);

  for(auto p : parents) {
    if(p->some_interface != nullptr) {
      (*p->some_interface->Method)();
    }
  }

  std::cout << "c1.something: " << c1->something << "\n";
  std::cout << "c2.something: " << c2->something << "\n";

  return 0;
}

Now if you're still with me and you haven't closed the browser tab in utter disgust let me explain all this and why I think it's easier to think about than polymorphism and inheritance (even if it's much harder to look at).

How it works

The parent is an ancestor, equivalent to an abstract base class in some ways. It holds pointers to all the interfaces that "subclasses" might have, but unlike abstract virtual functions, all of those methods do not necessarily have to be defined by subclasses. When calling methods on a parent, one simply checks if the interface has been defined first before calling the desired method.

parent's are "inherited" by subclasses such as child, but one could imagine step_child and adopted_orphan having a parent field as well. parent's are what are passed around because they expose the interface their children define.

Those definitions come in the form of C++ lambda functions ^{new in c++11!} because they are closures which can capture variables in the surrounding scope.

But they can just as easily be regular C function pointers just that that requires casting back and forth from void* to an explicit type:

// sorry that this example is so different from the one above but it's what i had on hand

#include <stdio.h>

typedef struct _interface {
    int(*CommonMethod)(void*);
} _interface;

typedef struct object_a {
    int prop;
    _interface iface;
} object_a;

typedef struct object_b {
    int different_prop;
    _interface iface;
} object_b;

void func_that_takes_interface(void* obj, _interface iface) {
    printf("%d\n", iface.CommonMethod(obj));
}

int object_a_common_method(void* obj) {
    object_a* a = (object_a*)obj;
    return a->prop * a->prop;
}

int object_b_common_method(void* obj) {
    object_b* b = (object_b*)obj;
    return b->different_prop * 2;
}

int main() {
    object_a a = {0};
    object_b b = {0};

    a.iface.CommonMethod = object_a_common_method;
    b.iface.CommonMethod = object_b_common_method;

    func_that_takes_interface((void*)&a, a.iface);
    func_that_takes_interface((void*)&b, b.iface);

    a.prop = 9;
    b.different_prop = 9;

    func_that_takes_interface((void*)&a, a.iface);
    func_that_takes_interface((void*)&b, b.iface);

    a.prop = 4;
    b.different_prop = 10;

    func_that_takes_interface((void*)&a, a.iface);
    func_that_takes_interface((void*)&b, b.iface);

    return 0;
}

It should be noted that C++ lambda's not only let you forgo this inconvenience but let you define a function body inline (which is what they were actually designed for...) but that goes against what I'm about to say next:

In either case, we can provide a regular function that takes an object receiver to serve as the function which satisfies a interface method. In this way, interfaces are not required to interface with a particular method that acts on a set of data but can be used when such situations arise that the sets of data, though with similar methods defined are of differing types.

Implementation details

std::function<> is necessary to hold pointers to lambda's. The one defined above is actually of type main::lambda<void()>.

Heap allocations are necessary even in this toy example.

The corresponding free functions are omitted for clarity above but are necessary nonetheless:

free_parent(parent* p) {
  if(p->some_interface != nullptr) {
    delete p->some_interface->Method;
  }
  free(p);
}

free_child(child* c) {
  free_parent(c->parent);
  free(c);
}

In Practise

I'm making a game. Every player, enemy, and npc has an entity struct field. That entity has interfaces such as renderable and updateable which are called by the game loop. It's working as intended so far. I need to add collidable and interactable next but I'm too busy being a nodev