Protobuf messages are gaining popularity across many domains. There are libraries like nanopb in C to support the protobuf encoding/decoding. However they are slow and also pose significant challenge in implementation. GoLang protobuf encoding/decoding is fast and provides support of unmarshalling data from JSON to proto format.
We will be creating a C shared library containing functions that read from json file, protobuf encode and decode.
Full Code can be found here:
brundhasv
/
GoC-Protobuf
Go functions integrated with C
GoC Protobuf Encoding/Decoding
Go functions for Protobuf Encoding/decoding integrated with C
Usage:
protoc --go_out=. Student.proto
go build -o student.so -buildmode=c-shared Student.pb.go student_en_dc.go
gcc -o student student_cgo.c ./student.so
Go program can also be run in standalone mode:
go run student_en_dc.go Student.pb.go
Explanation of Code here:
https://dev.to/brundhasv/go-protobuf-encoding-decoding-integrated-with-c-3ig0
GoLang Code
Lets start with Student.proto. I am using proto3 for this example:
syntax="proto3";
package main;
message Parents {
string dad=1;
string mom=2;
}
enum Gender {
Female = 0;
Male = 1;
}
message Student {
string name = 1;
int32 age = 2;
Parents parents=3;
Gender gender=4;
}
message Students {
repeated bytes StudentEntry=1;
}
As you can see above proto includes enum, nested proto and repeated bytes. Student represents a student's data, each student's data will be encoded and added to Students StudentEntry and all the encoded entries will be encoded again.
Below is input JSON file student_list.json which is used to populate proto fields. We have to use the same names and nested levels for the unmarshal of Json to proto.
{
"0" : {
"Name": "Elliot",
"Age" : 15,
"Parents" : {
"Dad" : "George",
"Mom" : "Elena"
},
"Gender" : 0
},
"1" : {
"Name": "David",
"Age" : 14,
"Parents" : {
"Dad" : "Keith",
"Mom" : "Sarah"
},
"Gender" : 1
}
}
Encoding Protobuf from JSON
To integrate with C, library C is used to convert GoData to C datatypes. Comment export is used to export functions to C lib.
We will create a new GoLang program file student_en_dc.go and add encode and decode functions to it.
Json file is input parameter of this function get_student_enbuf passed from C. This function returns encoded buffer and its length to C.
//export get_student_enbuf
func get_student_enbuf(student_json_file *C.char) (unsafe.Pointer,C.int) {
.
.
return C.CBytes(students_en),C.int(int32(len(students_en)))
First step in encoding is to Unmarshall JSON to proto. As there are more than one student entries, I have added entries using number keys. This cannot be directly unmarshalled, we would have to extract json data of each student and then unmarshal. I am first unmarshalling json file data to an interface. Below is the code:
jsonbytes, _ := ioutil.ReadAll(jsonFile)
var parsed_proto map[string]interface{}
json.Unmarshal(jsonbytes, &parsed_proto)
Then parse each interface to obtain student json data and unmarshall it into our proto. Aftermath proto is encoded as shown.
for i=0;i<len(parsed_proto);i++ {
selectedjb, err := json.Marshal(parsed_proto[strconv.Itoa(i)])
if err != nil {
log.Println("Error in reading json index",err)
return nil,0
}
var parsed_jf map[string]interface{}
//Each student data to proto
json.Unmarshal(selectedjb, &parsed_jf)
student := &Student{}
//Proto Encoding of student
json.Unmarshal(selectedjb, student)
Encoded data is added to StudentEntry of Students. Then Students data is encoded. We have the encoded buffer.
students.StudentEntry=append(students.StudentEntry,student_en)
}
students_en, err := proto.Marshal(students)
Decoding protobuf
Students is decoded first following all the student entries.
Function decode_student_enbuf takes encoded data and its length as input passed from C and returns decoded buffer to C.
//export decode_student_enbuf
func decode_student_enbuf(data *C.char,leng C.int) *C.char {
.
.
return C.CString(decoded_student)
}
Before decode, *char is converted to GoBytes and then unmarshalled to proto.
new_students := &Students{}
student_en:=C.GoBytes(unsafe.Pointer(data),leng)
err := proto.Unmarshal(student_en, new_students)
Decoded students data is used to obtain each StudentEntry and is decoded. Decoded student is appended to a string and is returned to C.
for i=0;i<len(new_students.StudentEntry);i++ {
new_student := &Student{}
err = proto.Unmarshal(new_students.StudentEntry[i], new_student)
decoded_student=decoded_student+"\n"+proto.MarshalTextString(new_student)
Building C-Shared file
Lets compile proto to generate Student.pb.go.Then build c-shared file.
protoc --go_out=. Student.proto
go build -o student.so -buildmode=c-shared Student.pb.go student_en_dc.go
student.so and student.h files are generated.
Following lines would be present in the generated header file. Golang function get_student_enbuf returns tuple(more than one return parameters), this is converted to struct when header files are generated as seen below.
/* Return type for get_student_enbuf */
struct get_student_enbuf_return {
void* r0;
int r1;
};
extern struct get_student_enbuf_return get_student_enbuf(char* p0);
extern char* decode_student_enbuf(char* p0, int p1);
C Code
From C side, generated header file is included. Students list JSON file is passed to function get_student_enbuf, returned encoded buffer and its length is populated in struct stu. The same are passed to decode function decode_student_enbuf, the returned decoded string is printed.
Finally both encoded and decoded buffers are freed.
#include <stdio.h>
#include "student.h"
#include <string.h>
#include <stdlib.h>
#include <stdbool.h>
struct get_student_enbuf_return stu;
int main() {
stu = get_student_enbuf("student_list.json");
int stu_list_len = stu.r1;
if(stu.r0!=NULL) {
char *decoded_stu = decode_student_enbuf((char*)stu.r0,stu.r1);
if(decoded_stu != NULL){
printf("%s",decoded_stu);
free((char*)decoded_stu);
}
free((char*)stu.r0);
}
return 0;
}
Compiling and executing C
Link with the created student.so, create object student and run the object.
gcc -o student student_cgo.c ./student.so
./student
name: "Elliot"
age: 15
parents: <
dad: "George"
mom: "Elena"
>
name: "David"
age: 14
parents: <
dad: "Keith"
mom: "Sarah"
>
gender: Male
Full Code can be found here:
brundhasv
/
GoC-Protobuf
Go functions integrated with C
GoC Protobuf Encoding/Decoding
Go functions for Protobuf Encoding/decoding integrated with C
Usage:
protoc --go_out=. Student.proto
go build -o student.so -buildmode=c-shared Student.pb.go student_en_dc.go
gcc -o student student_cgo.c ./student.so
Go program can also be run in standalone mode:
go run student_en_dc.go Student.pb.go
Explanation of Code here:
https://dev.to/brundhasv/go-protobuf-encoding-decoding-integrated-with-c-3ig0
Top comments (0)