DEV Community: Arthur Brainville

Mixing C++ with AMD64 (x86_64) assembly

Arthur Brainville — Sat, 09 Nov 2019 04:05:23 +0000

Lately, I’ve been dabbling into some “closer to the metal” kind of programming.

On most compilers (Visual Studio’s one for instance) it used to be rather easy to mix assembly code and C++ code using a feature called inline assembly where the ASM code will be put in a block (decorated with a special macro/symbol like _asm for instance), and when the compiler sees that, it will put the content of this “as is” inside of the compiled code.

Well, that feature don’t exist once you compile in 64bit mode, and also, mixing programming languages in a single file is a bit gross. Each time I see this, I think it feel rather ugly…

Instead, it would be good to separate that code into files, that will be assembled into object code you can link to your program as you do with C++ compilation units and static libraries…

So, how do you go about doing this, and doing it in a simple, and cross platform manner.

Well, first of all, there’s 2 things to consider when writing code in assembly.

1) The code you write obviously depend on the machine instruction set, since you are basically asking a CPU to do work, but you do it *instruction per instruction*. This means that you can only target one family of processors with that code. So when I say cross platform, I mean different rutnime on compatible machines. For example Windows VS Linux VS MacOs running on good old Intel (or compatible) chips.

2) Generally, you don’t just write instructions when writing assembly, you also write directive for the program that turn your code from human readable thing with letters and spaces and comments into the digital gibberish the CPU actually eats up to work. And there are multiple of them, and they aren’t really compatible with each other.

I was on a windows machine when I started experimenting, and saw that there was obviously the MASMoption, the macro assembler from Microsoft that was already installed on my box, as it is distributed with Visual Studio. Great… And I started playing with it, it’s not hard. I even got CMake to generate automatically a build system that work. But sadly these effort were vain as I quickly released that, I would not be able to use Microsoft’s assembler on Linux or a Mac for quite obvious reasons…

So, as always… Open-Source software win the day!

I was working on something totally unrelated that involved me having to build from source the OpenSSL library. When looking at the dependencies I needed to install on a bog standard windows box to do so, beside their horrendous build system written in Perl, one program stood up: NASM.

NASM is, my friend Google told me, the Netwide Assembler. It’s a BSD licensed Open-Source macro-assembler for 16, 32 and 64bit intel chips! Great news! It is cross platform and work on Linux and MacOS too (and I bet a bit more stuff!)

Also, I started googling around to see if CMake could auto-magically handle it, and indeed it does since a prehistoric version.

So, I got around to test that, and well, beside a tiny snag on MacOS that is apparently due to the way MachO object files differ from ELF files (a sad story of respecting standards about leading underscores) that is quickly fixed by telling NASM to prefix all global symbols with a _ character, I got a thing that just worked.

So, the result of this silly experiment is a demonstration repository that has nothing impressive to show beside how easy it works. It’s here, there’s C++ code, there’s ASM code, there’s a CMake project file that compiles everything together, the C++ code declare some extern "C" functions with names and interfaces that just happen to match the globally defined exportable symbols in the assembly code, and… voilà!

https://github.com/Ybalrid/cmake-cpp-nasm

Feel free to steal the literal 4 lines of configuration needed for this thing to work and go do some cool stuff!

SDL: not (just) a 2D graphics library

Arthur Brainville — Tue, 03 Sep 2019 19:03:04 +0000

My first real introduction with programming was with the C programming language, on a French website that was, at the time, known by the “Le Site Du Zéro” (think “The newbie’s website”).

At this time, I was in middle school, and I was pretty bored by it. It’s around that time that I started to have really access to the internet at home, and started spending a lot of time on it.

There were a few things behind this, mostly I was looking for trying to understand how video games works and how they were made. I was also trying to learn more about how computers work, without actually knowing what to look for, and I started dabbling into Linux in that time. This was around 2007.

This website had a well written, comprehensive, and actually fun “Introduction to C programming” tutorial. (They called them tutorials, but think of them more like courses…) Today, this website became “Open Classrooms”, a popular french MOOC platform, but the originals “tutos” are still up, and were (for some of them) improved, some turned into video courses.

This one was divided into 3 parts, one that is you “C programming 101” type of stuff, were they show you how to create variables, functions, if statements, loops, structures… This kind of things, the 2nd one is about more advanced things with pointers, memory allocations, arrays and whatnot, and the 3rd one is about showing you how to use a library, and takes example of the SDL to create a window, display things, and make a little sokoban game (with stolen Super Mario World sprites! 😉.

In this thing, SDL is described as being a 2D graphics library for games, and for years, I regarded the SDL as being just that. I have no clue if other people had this misconception, but when I look for things about the SDL, I mostly see things about making 2D games with it, or seeing it merely as a way to get a window and a screen and not much else.

But, oh man. It’s so much more than that.

The SDL name did make little sense to me at that time, but I never really thought about it. SDL is the S imple D irectmedia L ayer. A better way of describing the SDL would be a “cross platform API for doing most of the things you would want to do with an OS , including drawing windows, creating threads, loading DLLs, and pretty much anything.

I’ve learned recently that it was developed to be used as an aid for porting software from windows to other platform. The main idea being to support functionalities in a convenient package that are analogue to what was available with Microsoft DirectX. This was developed by former Loki Software employee Sam Lantinga. The business-model of that company was to port video games from windows to other platforms, and mostly Linux.

The SDL works on pretty much anything you would want to use. Today, for me it looks more like a fundamental building block for multimedia programs (video games, and other interactive things with custom graphics), and without having to worry too much about the platform you are running into.

If you use the SDL, anything you do through it will work on a computer running Windows, MacOS, Linux, flavors of BSD, obscure little things like Haiku (Haiku is awesome. This is a really interesting operating system project), any kind of smartphone being Android or iOs, and probably much more things like video game consoles. All these platforms are available to you without having to change your code too much (not at all most of the time).

Since version 2.0 came out, the SDL is licensed under the zLib license. Before that, SDL was LGPL. LGPL is not a problem in commercial, closed source software (like video games usually are) when the library is dynamically linked to the program (for example, on Windows, the library is a .DLL file distributed alongside the binaries of the game.) This permit you to recompile and change the library, and the developers of the game needs to make the source code of the version used with the game available to you, including any changes they may have done to it, and under the same rights. This can make some company’s lawyers uneasy. the zLib license is a much shorter, much straightforward license that pretty much says “do whatever with the code of this, and credit about it is appreciated”

Also since version 2.0, the huge feature list of the library was expended further. The impressive feature list can be read here.

I spent quite some time misrepresenting what the SDL was, thinking that it was just a lousy 2D graphics libraries with a software renderer, and not the impressive tool that it actually is.

Few years ago, while searching content from Vavle about their SteamVR system, I stumbled across this talk by Ryan C. Gordon, a prolific freelance programmer that, if you ever ran a video game natively on Linux, there’s a pretty high chance he helped make this a reality. I really invite you to watch this, and appreciate all the hard work this simple C library can do for you in just a few lines…

https://www.youtube.com/watch?v=MeMPCSqQ-34

As a side note, I really should plug a little project a good friend of mine started some time ago (and which I contribute to, because it’s awesome) to make a safe , C++17, RAII wrapper around most of the useful bits of SDL for 2D/OpenGL/Vulkan programming. Check it out here: https://github.com/Edhebi/cpp-sdl2

No nonsense networking for C++ : Introducing kissnet, a K.I.S.S. socket library!

Arthur Brainville — Tue, 04 Dec 2018 21:02:36 +0000

this was originally published on blog.ybalrid.info

Sometimes I wonder why some things are inside the C and C++ standard libraries, and some aren’t.

As far as I can be bothered to read the actual “standards” document (that are mostly written in legalise, not in understandable english for you and me). Theses languages are defined against an “abstract machine”, and the actual real-world implementation of them, on computers that actually exists, should follow the behavior described for that thing, modulo some implementations details.

Beside the specific case of having theses languages in a “free standing environment” (meaning that the code written isn’t actually relying on being executed inside an operating system, but is directly running on the bare hardware), It seems that some notions, like the OS having a “filesystem” where textual paths can points to files that can be opened, read and modified, is pretty standard.

What is strange about that is, if the notion of files and file systems are part of the standard library for both C and C++, (and were present since the beginning) networking sockets doesn’t exist in both languages. And C++ gained the notion of creating multiple threads of execution, and manipulating them inside it’s standard library only in 2011.

All of theses concepts : files, threads, and sockets, are Operating System specific constructs. Opening a file on Linux is fairly different that opening a file on Windows for example. But the standard library offer a single, unique, and portable way of doing so.

These three things are present in all operating systems in use for the past decades now (since the 70’s at least?). I find it strange that the C standard library only includes files. Since C++ now also has threads, I will consider this a non-issue. So let’s talk about the other one…

A song of files and sockets

I would like to take some time to discuss writing some low-level network code in C++. The current interface to get data to and from a network we know today are using a notion called sockets.

For lack of a better analogy, a socket can be thought as some kind of “magical file” that, when, written into, will send the bytes that has been written to on the network, and when receiving bytes, they will be accessible by reading said file. This notion comes from the UNIX world, where everything is effectively a file. And nothing is wrong with that, it’s actually a really simple, straightforward way of doing things.

Most of the operating systems that matters today are using this analogy. When I say OSes that matters today, I’m thinking of both the modern UNIX derivatives (Linux, macOS, and the rest of the BSD family). And Microsoft Windows.

The Windows socket API has been mostly borrowed from BSD anyway, if you remove some oddities like a few renamed functions, a few changed data types, and the added initialization procedure, Windows sockets are equivalent of sockets you have on Linux.

100% non-standard code

But, none of this exist inside the standard library of these languages. When you are doing socket programming on Linux, you are not calling functions of the library, you are performing Linux Kernel System calls, and you are dealing with file descriptors and bytes.

On Windows, you are calling part of the Win32 API (called WSA for Windows Socket API). This situation is unfortunate, because it means that I , as a C++ programmer, I need to make sure that my code will work both under Linux and under Windows. There’s no one single networking API that I can use everywhere without thinking about it. Sure, it’s 80% similar, or maybe 90% similar, but still, if I need to put

#ifdef WIN32

in my code for something as fundamental as sending bytes to another computer in a network in 2018, we are doing something wrong.

Moreover, all theses OS-level API are implemented in C. Not C++. This means that everything you have are functions and structures. When describing socket configuration, you are filling structures and passing them to functions. When you have to reference a specific socket, you need to keep a little token and give it to a functions. When you need to read data, you need to have a buffer with the correct amount of bytes ready and give a pointer, alongside a variable containing the size of said buffer to a function, and make sure that you don’t mix them together.

Basically, you are doing 1970’s level computer science. This is fine for the lowest level code out there, but not for the code of an application.

Unnecessary added complexity

There are solutions to solve this, and some that are even well advanced in the path of getting standardized inside the C++ language, like Boost.Asio. But, In my humble opinion, there’s a fundamental problem with Asio itself: it is As io.

For those who aren’t familiar with Asio, it’s name stands for “ as ynchronous i nput and o utput”. It’s a library, a good library for what it’s name stands for: doing intput and output in an asynchronous manner. For doing this, Asio as multiple contexts and constructs to deal with multiple threads and non blocking calls, and who executes them, and when they are executed.

The problem is: If I just want to connect to the network and exchange data, do I need to worry about io-context and handlers, and executors? Probably not.

Keep It Simple, Stupid.

In the C++ world, we struggle at keeping simple thing simple. The collection of libraries from the Boost project is one good example. Don’t get me wrong, these are high-quality, peer-reviewed C++ libraries. They are good code written by smart people, with the seal of approval by other smart people.

They are a demonstration of what you can do when you want to push the language forward. They contains a lot of useful pieces of generic code that you can reuse. A lot of really important and useful things from Boost finally landed inside the C++ standard library since 2011, like smart pointers, chrono, array, lambdas, and probably a lot more things like that are going to jump from existing on boost, to being implemented inside the standard.

And, if today you ask for a recommendation of something to use to do networking code in C++, I’m almost sure that you’ll get pointed to either Asio, the version of Asio inside of Boost, or, the Networking TS (addition to the standard that will probably land in a future version of the C++ language) that is… Basically based on Asio.

As you can guess… It’s not like I don’t like Asio, I find it genuinely interesting, and potentially useful. But I’m unsure it is the thing that I want to see standardized.

As stated earlier, if you’re just going to do some TCP/UDP communication, Asio comes wrapped in unnecessarily complexity.

Moreover, your OS comes with a socket API, but it isn’t super convenient to use in C++, and it’s not portable without doing the ugly

#ifdef

preprocessor dance.

Few months ago, I was thinking about this situation, and thought why not just wrap the os library in a nice C++17 interface?

Introducing kissnet

Kissnet is a little personal project I started during the summer, and that I tweak a little from time to time, it’s mostly for fun, but I feel like some people could have some use for something like this.

The design goals of kissnet are pretty straightforward:

Be a single header library, nothing to link against
Be a single API for all supported operating systems
Use C++17 std::byte as a “safe” way of representing non-arthmetic binary data
Be the lighthest/thinest possible layer on top of the operating system as possible
Handle all (or most) of what TCP and UDP sockets can do in ipv4/v6
Don’t require the user to worry about the endianess of the network layer.
Just transport the bytes, and do nothing else
Hide os-specific weirdness
Optional exception support (You can chose to replace throwing exception with the program either aborting or calling your own error handler)
stay simple

Kissnet only implement 3 kinds of objects:

A socket class. The behavior of the socked is templated around the protocol used (TCP vs UDP) and the version of IP used (ipv4 vs ipv6)
An endpoint class that permit you to specify a host and port as just a string and a number, or a “hostname:port” string.
A “buffer” class that is just syntactic sugar around an std::array

Buffer are for holding received data, buffer know their own size, and can read the correct amount of bytes for you. Kissnet doesn’t care what data is sent, it’s not kissnet job.

Sockets are non-copyable (but movable) objects and they have the typical operations you can apply on socket implemented (bind, listen, accept, connect, send, receive).

Kissnet automatically manages the initialization/de initialization of the underlying socket API if needed (like on windows). This is done by exploiting the reference-counting of std::shared_ptr<>. This is the only overhead on top of holding a socket “file descriptor” ( = an simple integer variable).

I’ve only used kissnet in a couple of toy programs and not in a real project, however, I already think that I prefer this simple, down to the metal but yet type-safe and cross platform library as using something like Asio. Asio feels like using a bazooka to do kill a fly. I’ve heard a few opinions going the same way as I do. This is why I’ve put this little experimental project on GitHub, under a permissive MIT license.

Why glTF 2.0 is awesome!

Arthur Brainville — Mon, 16 Jul 2018 13:40:58 +0000

originally published here : https://blog.ybalrid.info/2018/07/16/why-gltf-2-0-is-awesome/

There’s one single thing that I find truly frustrating when dealing with multiple 3D-related software : making them exchange 3D assets.

You don’t have much warranty that what has been put out of one software will look the same into something else (e.g. a Game Engine. You may work with meters, and find out that Unreal works in centimeters. They could use different conversations for texturing, material definitions may just “not work”…)

There’s scale issues, animation issues, texture binding issues, material problem in general.

All of this is generally dealt with a huge and horrible mess of import/export scripts. And the more of these you need to run in your toolchain, the worst it gets

Still, there are a number of file formats that are more or less standardized in this industry, but none of them does really fit all the use cases for real time rendering/video games (and by extension, VR).

Most of them are way better suited as “authoring” format: Files for 3D modeling programs, like Autodesk’s FBX or Collada from the khronos group. They are complex, and they generally are too-flexible in how they can be implemented and how they can represent a specific thing, and just contain “too much data” for what you would want to use from a programming stand point.

FBX seems to be a de-factor standard, especially in the video game industry, because of the predominance of Autodesk software as modeling animation tools. When they aren’t using 3Ds Max, they are using Maya… ! ;-)

Generally, what you would want to do is to transform theses assets into serialized binary files that can be loaded quickly into a game engine. This is generally done on a per-engine basis. For what I can tell, the general workflow of “putting thing inside Unity” generally involve putting (for example) .FBX files into the project directory, and when Unity will build your game it will convert every resources in formats optimized for the target platform.

Still, this approach of reinventing the wheel and reinventing one “transmission format” for each and every target application of 3D assets doesn’t seems to be the right thing.

In the 2D world, this problem was fixed when the whole industry basically standardized around the JPEG file format for 2D pictures. Evey single camera out there is (special cases or advanced users aside) outputting JPEG encoded and compressed image files.

Obviously, a JPEG file doesn’t have all the details that a Photoshop “project” (a PSD) file would contain, but JPEG is not intended for authoring images, it’s intended for sharing them.

What we need is a “JPEG for 3D”, and that’s exactly what the Khronos group, with an open community effort, developed, and released as the “OpenGL Transmission Format”, and version 2.0 of the specification was released in June 2017.

This format permit to describe 3D objects (and even whole scenes with lights and camera, and animations) in a standard format. A contrario to Collada for example, there is no room to interpretation, and not 2 ways of describing the same thing in glTF

glTF represent model data as simple binary buffer. “Accessors” permit to interpret that data contextually. This is particularly adapted to loading theses files in OpenGL, where you could load each buffer into the GPU with glBufferData, then parse each accessor with glVertexAttribPointer to bind in to the location of each vertex element you have in a buffer.

glTF is based on a simple scene graph description, written in JSON, and a number of raw resources that are the binary buffers and texture data as images.

The standard is generally easy to understand in it’s plain text, but some of the notions used are simpler to understand with some drawings. Thankfully, there’s this awesome “What the Duck is glTF?” on the official repository:

This poster describe most of what you need to understand about the glTF file format

There are currently a ton of software, tools and libraries that now support glTF. and I hope they will be even more in the future.

This format is developed by the khronos group, alongside an official implementation of an exporter for blender, and it already has a lot of support form the industry

The glTF NASCAR jacked has some really imporant sponsors!

I don’t think I really need to develop more why I think every effort going towards having more support for glTF will be beneficial for the whole industry/comunity, but supporting glTF today should be your go to option.

Juan Linietsky, main Godot developer actually written a lengthy article comparing multiple format, concluding that we should all support glTF2, and I cannot agree more!

In my next post, I will relate what it took to write a glTF import plugin in C++ for the popular open-source Ogre rendering engine. It is currently a work in progress project, but It starting to become usable, and can be found here

Install and run SteamVR on ArchLinux (for using an HTC-Vive) and do OpenGL/OpenVR developement

Arthur Brainville — Tue, 20 Mar 2018 10:33:54 +0000

So, I recently had the chance to try out an HTC-Vive on a Linux machine. Naturally, I installed Arch on it !😉

The installation is pretty straight forward, but there are some little catches if you want to do OpenGL development on Linux With OpenVR (OpenVR is the API you use to talk to the SteamVR runtime.)

SteamVR has a Linux beta since February 2017. They also announced that the SteamVR runtime itself is implemented with Vulkan only.

First thing first, I am unaware if it’s currently possible to update the firmware of the base station on Linux, The setup I’m using is also running on Windows and the firmwares where already updated on that platform. That’s one thing I can’t tell you about.

So, to get started, you will need to have Steam running on your machine. ArchLinux is now a 64bit only distribution, Steam is a 32 bit only program. So, if it’s not already the case, you need to activate the [multilib] repository for pacman. (just uncoment the lines for it in /etc/pacman.conf)

First thing first, to do VR, you do need a good GPU. Here I’m running an Nvidia GTX 1070 with the proprietary drivers. You can use an AMD one with the latest version of the mesa drivers apparently. I don’t have access to any modern AMD graphics card so I can’t tell.

Then, you will need to install the following packages:

steam
lsb-release
your graphic’s driver packages in 32 bit (lib32-nvidia-utils, lib32-libvdpau)

Once you have installed Steam, launch it, connect or create an account, and install the SteamVR package, then you want to turn on the beta:

Once you have steam VR on your machine, and before you plug the Vive on the computer, you will need to install some udev rules to permit the app to access the device directly.

Create a file /lib/udev/rules.d/60-HTC-Vive-perms.rules and write the following content in it:

# HTC Vive HID Sensor naming and permissioning KERNEL=="hidraw\*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="0bb4", ATTRS{idProduct}=="2c87", TAG+="uaccess" KERNEL=="hidraw\*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="28de", ATTRS{idProduct}=="2101", TAG+="uaccess" KERNEL=="hidraw\*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="28de", ATTRS{idProduct}=="2000", TAG+="uaccess" KERNEL=="hidraw\*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="28de", ATTRS{idProduct}=="1043", TAG+="uaccess" KERNEL=="hidraw\*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="28de", ATTRS{idProduct}=="2050", TAG+="uaccess" KERNEL=="hidraw\*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="28de", ATTRS{idProduct}=="2011", TAG+="uaccess" KERNEL=="hidraw\*", SUBSYSTEM=="hidraw", ATTRS{idVendor}=="28de", ATTRS{idProduct}=="2012", TAG+="uaccess" SUBSYSTEM=="usb", ATTRS{idVendor}=="0bb4", ATTRS{idProduct}=="2c87", TAG+="uaccess" # HTC Camera USB Node SUBSYSTEM=="usb", ATTRS{idVendor}=="114d", ATTRS{idProduct}=="8328", TAG+="uaccess" # HTC Mass Storage Node SUBSYSTEM=="usb", ATTRS{idVendor}=="114d", ATTRS{idProduct}=="8200", TAG+="uaccess" SUBSYSTEM=="usb", ATTRS{idVendor}=="114d", ATTRS{idProduct}=="8a12", TAG+="uaccess"

You can now open SteamVR, and you should be prompted to run the room setup program to configure your “play space”

Then, if you want to use OpenVR to render to the Vive with OpenGL, you will need a Vulkan runtime and development package. For this, you need to install the

vulkan-devel

package.

You may also need to launch your programs under under the steam runtime, this is done using this script :

~/.steam/steam/ubuntu12\_32/steam-runtime/run.sh ./my\_steamvr\_app

And that should be about it. Programs that link against the OpenVR API will “just work” now.

The info in this article comes from my own experience, and this document from Valve : https://github.com/ValveSoftware/SteamVR-for-Linux

Annwvyn, my game engine is now “officially” compatible with Linux with OpenVR. !😉

I would be really happy if it was possible to use the Oculus Rift on Linux, yet, there aren’t any good solution right now. Oculus froze Linux development effort years ago, so we aren’t going to see an official SDK any time soon. The OpenHMD project has made “some” progress” but nothing really usable for now.