DEV Community: Atsushi Eno

moving

Atsushi Eno — Sun, 10 Mar 2019 10:08:42 +0000

I need easy and kind-of-safe options to store and embed images on the posts. Therefore I'm moving the blog to https://atsushieno.github.io/ . I'm using jekyll + prose there and prose is far from an ideal solution especially wrt images, but I can kind of hack around it. So far, newer posts will appear there, unless I dump the new solution away (I'm still kind of indeccisive yet).

ADC2018, SOUL and APLs

Atsushi Eno — Thu, 22 Nov 2018 11:50:52 +0000

ADC2018

When I kind of decided that I leave Xamarin development job and dive into audio development world in early summer, I had one thing to do in mind: join Audio Developers Conference (ADC) 2018.

The two days conference (three days if training day is counted) is over, and I was quite happy there. There was an announcement about a new language called SOUL, the SOUnd Language. You can find the session recording on Youtube.

https://youtu.be/UHK1QQM0bnE?t=910

These days I had been checking various music and audio related software libraries and tools to write many blog entries for my experimental audio tech/library/tools Advent Calendar in Japanese, and reading a handful of text about this kind of stuff for it. I think I understand how awesome this SOUL stuff could be (as a skeptical developer I would keep saying that it was still just an announcement) and thought I should describe why. It is partly translated from what I have prepared in Japanese, even before ADC had started.

I have to say, I have never been familiar with audio development. I joined ADC2018, while I didn't know that it is actually mostly about JUCE. I kept telling friends like "I will be joining ADC to possibly find a next job there", but I was not sure what it is like. I was not even serious about joining it ("I would join it, but I'd quickly give up if something more fun thing happens" kind of). There are things that are likely obvious to those audio devs attending there and yet I didn't know. I am never a dedicated C++ programmer. But ADC was still fun, I learned a lot of things.

So, what is APL like?

SOUL is meant to be a new APL（audio programming language） and further, platform. APLs are languages, not just a single language (and here it does not mean "A Programming Language"). Take it like VPLs-alike (visual programming languages).

They designed to process audio and compose (or more precisely, "create" ?) music or realize sound effects. Ideally, they are designed for non-programmers (or maybe "not very advanced" programmers).

They are actually language-based tools rather than language specifications. Examples of APLs are: Csound, ChucK, Pure Data, Alda, Faust, Tidal. What is interesting there is, they always define their own languages. Actually some of those languages are simply based on Lisp or Scheme and it is argurable that they are under their own "language", but let's skip that so far, they ultimately end up with their own ecosystem.

An interesting aspect of those languages is that they never make use of virtual machines like Java or .NET. Describing music or sound effects would be very useful especially in apps like games, and there are many apps or games written in Java or .NET. Why should we learn a completely different language? Can't they just be some audio libraries like raw-audio libs or raw-midi libs so that we can arbitrarily manipulate those sound objects...?

However, those APL designers have reasons to do so. Interestingly some of those are described in academic papers. Among those papers I pick up an interesting one from Andrew Sorensen, regarding his language Extempore. It is described from very primitive level, examining every possibility to achieve minimal latency.

https://openresearch-repository.anu.edu.au/handle/1885/144603

Extempore is a language for "live coding" (he calls it a "cyber-physical" programming language). You can find it from this keynote recroding from OSCON 2014:

https://www.youtube.com/watch?v=yY1FSsUV-8c

Audio processing use cases

Typical jobs that APLs deal with are often about "realtime" processing of audio. What are such jobs for example?

Think about a virtual piano app. You press a key, and the app reacts it by sending out piano sound to audio output. If you pressed a key and it took one second, then you don't want to use it.
MP3 players decode *.mp3 files and send the result to audio output. Time of decoding is usually shorter than raw PCM playtime, but if the output was flaky then it's useless.
DAWs (digital audio workstations) are used to compose music in mutiple tracks. Depending on songs, it might have to send audio outputs as well as MIDI outputs in sync. If they are out of sync, the resulting music does not sound right.
Think about live performance using digital instruments (like the OSCON keynote video), optionally with visuals synchronized. If the app freezes even very short time like less than 0.1 sec, it is still problematic.

Those tasks might not necessarily be "realtime" in the context discussed here, but those tasks are under tight requirement on "precise" processing time.

Realtime requirement

So, it seems precise processing time is important. But how precise they need to be? How "realtime" should they be? 1 second delay is obviously wrong. 100 milliseconds? 10? 1? ... It's not obvious to us anymore. Similar realtime-ness is required in 3D animation frame rates in VRs e.g. 60fps, 120fps... Audio is similarly constrained, it is said that if latency becomes like 20 milliseconds of delay then some people would notice.

I should make one point clear; realtime processing is NOT about high-performance processing as in "most efficient performance". The requirement is: a recurring task must be always invoked and run within the expected time frame. It is in a sense completely opposite of high-performance computing.

In modern general computing environment, there are preemptive multitasking (processes and/or threads) that are managed by the operating system. The Sorensen paper discusses other possibilities (manually-cooperative multitasking) and they are real on some embedded environment, but for APLs it would be mostly preemptive world. (Devices like ROLI Blocks and Littlefoot compiler might be the other way.)

In any case, what is important here is that such a realtime app needs to run on a raltime thread (or process) which needs to be supported by the OS. General threads don't have such guarantee that it must be invoked in "timely" manner.

And virtual machines have further problems. Namely, garbage collectors "stop the world" (including app threads) to collect unused memory blocks, and JIT compilers compiles VM code to the actual machine code at run-time. They result in uncountable delays.

Sorensen mentions SonicPi on Ruby, Gibber on JavaScript, Impromptu on Scheme, and Overtone on Clojure as examples.

Therefore, languages like Extempore, avoids those problems by designing their own language to generate native code statically, and require explicit memory allocation. These days LLVM-IR is the common code generator solution to them. Probably Lisp/Scheme parsers were their frontend solution too, that's my guess.

As of 2018 we, virtual machine based developers, would have some words on that premise (e.g. we have AOT solution), but let's put it aside. Today I'm more interested in what SOUL provides.

Language runtime and sound server communication

One interesting point that Sorensen made in that paper was that a full stack solution vs. half stack solution for an APL - it is still possible to write client-server system, and it can be either intra-process or inter-process. So, instead of implementing everything in one single language and framework, it is possible to use multiple languages and frameworks, implementing each interested parts and cooperate together.

For intra-process design, there would be a realtime sound server implementation and client language bridge by FFI. According to the paper, ChucK, Impromptu, and Fluxus are based on them. For inter-process model, SuperCollider is an example. Extempore is based on inter-process model too, at this state (the paper says it's been inbetween those two models).

the Language Barriers

Leaving Sorensen paper aside, I have been always interested in having some sound objects model that can be manipulated in C# and Mono. I had almost no interest in whatever .NET Windows people had created. Windows-only stuff will just die soon. I'm only interested in cross-platform solution.

There are still some hackers who created portaudio bindings or rtaudio bindings. I have my own binding for libsoundio, but there are other people who did it too. There is Web Audio API in Javascript world, and at ADC2018 there was a session about bringing audio API into C++ Standards ("std::audio"). There is no such thing in .NET. They are years behind C++ or Web.

Anyhow, the next step to raw audio, what I wanted was to have some common sound object models that those APLs could possibly share. Right now they are living around Babel tower, but some commonplace could help improve the situation towards cross-language solution.

Then I can instantiate audio output tracks and play in timely manner, coordinating them all, just like DAWs do. I have my own MIDI player and even text macro compiler to generate MIDI files, so I had some foundation.

I had some look at a couple of APLs, and I always end up to find no general music solution there. Their outputs are specific to some genres of music. It's not what I had seen back in 20th century in Japan - people use FM synthesizers and MIDI instruments for various kind of music, even with limited expressiveness.

My understanding is that there can be unlikely good language. Instead there could be common sound platform among any new possible languages.

Another concern I had was that I should probably write everything in C (or at least provide C API). My primary language was C# (although I'm practically free to choose any language now that I'm independent) and C# still has many developers, but if it has technical difficulty (like discussed above) then it is just a bad choice on technology.

The client-server model on that Sorensen paper was then likely the next thing I could try i.e. design primitive sound object system and manipulate them privately via C#.

To be honest, I don't really care much about audio latency. But I would mind if I were going to perform some sort of live coding, and that's still a possible future.

where SOUL takes place

SOUL announcement just appeared when my idea is like that.

(I have no images storage so I put them on Google photos, that you have to navigate to them...)

This is a screenshot from the video streaming. You can use any language to write a programm that manipulates SOUL API, which is likely something like OpenGL Shader. The SOUL language sources are compiled into some native code using LLVM IR and run by JIT.

What's good there is that the sound platform itself does not give up audio latency. It actually aims to provide better solution, by providing chance for hardware-accelerated processing of the sound platform:

Hardware driver accelerated programs run on better (less) latency. (Of course there are some business opportunities that the SOUL language developers could aim at.)

Yes, JIT is something that those APL designers worry about, so as I assume in theory. The solution here is similar to how shaders are compiled i.e. GPUs. GPUs are popular, but for this SOUL devices we would be mostly skeptical. That may or may not be a matter of time, I have no idea now.

Thoughts

My concern on SOUL is mostly about compilation pipeline, especially on how instant the turnaround time would be. If generating piano sound from keypress is somewhat slow, then it may not be a solution.

Extempore, as a live-coding ready language, compiles their XTLang sources into LLVM IR so that when it is invoked it is instantly executed. But it also involves compilation, which those language users would pre-compile on their editor. Compilation should still take some time, unless the language itself is not very expressive.

SOUL itself is not likely highly expressive (it does not seeem to aim to be a universal language) so the client languages (for me maybe C#) have to generate fairly complicated code.

But since there is no actual piece of code out yet, we'll see what happens. Even if it was just a vaporware (which I guess highly unlikely), it is still a great idea that could be accomplished by anyone so far.

Fluidsynth hack continued: Oboe driver

Atsushi Eno — Thu, 15 Nov 2018 15:44:01 +0000

Updates since the last effort

It was two weeks ago that I finally got a practically working fluidsynth port with OpenSLES driver support. It was after big restructuring on the build system (building glib is a complicated task, and my build setup was messed due to Android NDK evolution). And by the time I got back successful builds there is already a new "better" audio driver Oboe (which is modern and ready for low latency than before).

And here you are, the Oboe driver is done: https://github.com/FluidSynth/fluidsynth/pull/464

This post is a handful of followups on the new Oboe driver implementation. There were many traps that drugged me on the ground...

Mixing C++ and C

If you are not familiar with CMake, like me, you'd find it problematic on finding the right way to not let CMake ignore c++ sources. You need CXX in the add_library function call explicitly, which takes a list of languages to use for compilation. CMake silently ignores your *.cxx files, even if you clearly specify them in the source list.

(historical) Oboe needs Android NDK r17, while Cerbero specified r16

It is now only a past issue, but Cerbero android.config specified Android NDK r16 explicitly. It was upgraded to r18 about 5 days ago.

NDK r16 lacks some required constant definition in AAudio API, which results in Oboe build failure. Therefore it is mandatory to use NDK r17 or later anyways.

I ended up to hack around Cerbero sources (again!) - this time it was trivial and that particular AAudio issue was resolved. But then another problem showed up...

gnustl and libc++

Since Android NDK r17, it's moving away from gnustl and started using libc++ in clang. And with r18 gcc and co. are totally gone, meaning that there is no gnustl anymore.

I have been using Cerbero build system to get a working glib build with all the dependencies. Unfortunately, their build script still specifies android-16 (JellyBean) in config/cross-android-*.cbc, and that triggered some problem. Let's see NDK r17 release notes...

libc++ is now the default STL for CMake and standalone toolchains. If you manually selected a different STL, we strongly encourage you to move to libc++. Note that ndk-build still defaults to no STL. For more details, see this blog post.

also...

The platform static libraries (libc.a, libm.a, etc.) have been updated.

All NDK platforms now contain a modern version of these static libraries. Previously they were all Gingerbread (perhaps even older) or Lollipop.

Prior NDKs could not use the static libraries with a modern NDK API level because of symbol collisions between libc.a and libandroid_support. This has been solved by removing libandroid_support for modern API levels. A side effect of this is that you must now target at least android-21 to use the static libraries, but these binaries will still work on older devices."

What makes things complicated is Cerbero android build which in general (namely android/external/cerbero/packages/base-system-1.0.package) specifies 'gnustl' as part of the dependencies. gnustl is the standard C++ STL implementation library from GCC. See the first quote from NDK r17 release notes. They now have switched to libc++ and "strongly encourage" to move to libc++...

What happens if we mix uses of both? Some "unresolved reference" errors due to lack of either at linking.

Unfortunately, gnustl is widely used among many recipes and that should be handled by whoever is involved in Cerbero well (which is hopefully not very complicated).

solution: two shared libraries

Compared to the problem above, it is much easier to deal with, but another minor problem with Oboe was that it requires C++ build support. And that slightly messed up existing CMakeLists.txt.

To sort out those issues, I ended up to build a C-based oboe shared library liboboe-c.so and libfluidsynth.so.

liboboe-c.so: based on C++, links libc++
libfluidsynth.so: based on C, links gnustl

While I had been trying to get this working, there were some new commits in Cerbero and it had moved to NDK r18. But since gnustl is still in use, the symbol expectation mismatch still happens. I decided to keep this liboboe-c.so until it can go totally unnecessary...

(And in the future when Android NDK r19 lands, it will also migrate from gold linker to lld, which could bring in other kinds of problem.)

(I might add more lines later, but so far I'd publish this first to make it visible on what I've been doing regarding them.)

Fluidsynth 2.0.x for Android

Atsushi Eno — Wed, 31 Oct 2018 17:18:21 +0000

TL;DR

My fluidsynth Android port is live again, caught up with the latest Fluidsynth development (2.0.x).
There is Java version of fluidsynth based on JNA.
There is native asset loader now.

Recap from the last post

Back in March, I wrote a post on my Fluidsynth port to Android. Since then I spent some time or have been doing busy on other stuff (such as the latest TechBookFest5 as the management team). But now that I'm not employed and have a lot more flexible time, I could resume my Fluidsynth port as well as many other music related software I had been working on. Anyhow...

To recap the latest state, I had an Android port of fluidsynth, using a fork of Cerbero, organized by my own module called android-fluidsynth. The build is complicated so that almost no one could use my outcome, and the original Fluidsynth had moved to github with evolving changes e.g. migration to CMake build system. The CMake build system on Cerbero didn't work nicely for Android.

Resurrection and restructuring

To make things worse, my android-fluidsynth build got broken when Google updated Android NDK to whichever version I don't even remember - I had to manage Android NDK support in my Cerbero fork tree, and it's been somewhat annoying. And NDK had various changes that make updates messy - aarch64 support, switch to unified headers (that means, file lookup paths change), and the default toolchains switch to Clang (and gcc getting obsoleted).

So, I began with refactoring the entire build dependencies. This was what I had in the beginning:

atsushieno/fluidsynth-midi-service
  atsushieno/android-fluidsynth (submodule)
    atsushieno/cerbero (submodule)
      atsushieno/fluidsynth (git checkout)

My cerbero changes were for 1) NDK lookup changes, and 2) additional fluidsynth build recipe. I decided to remove fluidsynth build from there. That means, cerbero still builds glib (still required as a dependency) but I could just use the outcome and reference it from another CMake-based fluidsynth build (which is the current build system).

(I could even package those native binaries for glib, but now it is just to checkout and run cerbero build, which is maintained by their own team, which is simple enough for me.)

To align with that, I ended up to create another fluidsynth fork which is based on the latest original tree (Fluidsynth/fluidsynth on github) and ported my opensles additions (which needed some additional changes, but that's not important here).

Along with this way, I could significantly reduced my dependency tree:

atsushieno/fluidsynth-midi-service
  atsyshieno/fluidsynth-fork (new, submodule)
    GStreamer/cerbero (on-the-fly checkout at build time. Not on github)

... and build it again(!)

Once I could submit a PR to the original Fluidsynth repo and got accepted, then the dependencies will look even simpler, but let's think about it later.

Kotlin based application

In the earlier days, I had been using Xamarin.Android as the primary app development framework for this fluidsynth port. However after I quit the development team, it became impossible to keep using the IDE on Linux. I still continue the development, but it became quite tough without IDE.

After many thoughts I ended up to start porting part of my C# app to Kotlin, using JNA for my NFluidsynth replacement. The actual API binding is almost automatically generated using JNAerator. Since I didn't want to mess with JNI, it was a big help to me. (I was a BridJ contributor in the past, but now JNA looks easier to depend on.)

The ported app is far from a feature parity port, but at least it is enough to dogfood the fluidsynth Android port. In the end there will be fully functional MidiDeviceService implementation.

Custom SoundFont loader for Android Assets

One of the big API changes in Fluidsynth 2.0 was a completely rewritten custom SoundFont loader. In Fluidsynth 1.x we had to completely implement SF loader that does not only provide custom stream processor but also had to build the entire SoundFont structure, which makes it almost impossible to customize. Fluidsynth 2.0 offers a new way to provide a set of custom "stream callbacks" (for open/read/seek/tell/close) that lets us implement our own stream loader.

My previous port had some changes to provide custom stream loader which had exactly the same purpose, so I could totally remove those changes. Instead, now I (kind of) have to provide custom SF loader functionality for Android Assets that aligns with the new way.

There are two options:

implement custom callbacks in native (as in NDK manner) API.
implement callbacks in the wrapper languages (C# for Xamarin.Android, Java for ordinal Android developers).

What makes the former (native API approeach) annoying is that the NDK Assets API requires JNIEnv and jobject to acquire AAssetManager. It was especially annoying for my new Kotlin-based app. I ended up to add Java_fluidsynth_androidextensions_NativeHandler_setAssetManagerContext() function in the Android port. (It is nasty especially because any Java code that tries to use this API needs to have NativeHelper class in fluidsynth.androidextensions package, that I offer in my Kotlin app. I even renamed my Kotlin app package from name.atsushieno.fluidsynth to fluidsynth to make it less nasty...)

The second approach (to write callbacks in wrapper languages) looks viable, but it needs some special care about GlobalRefs for the Java objects that provide those callback functionality - whenever GC moves your method references, the Asset SF loader crashes! I haven't resolved that issue with my JNAerated API yet (C# version works as I made it "pinned" with GCHandle). Native interop is a wild where you're killed without sufficient crash information or chance to debug...

Building a debuggable libfluidsynth.so

Even after I got a "successful" libfluidsynth.so builds, it never worked successfully. The longstanting problem was an unexpected SIGILL. This issue was even registered at luckier state than at first, as it has a debuggable binary(!)

There had been wide variety of chances that libfluidsynth.so was built without debug symbols. For example, you have to...

explicitly specify debugging options such as -Denable-debug=on and give additional -fsanitize=undefined and -fsanitize-trap=undefined flags at CMake,
make sure to pass -g or those -fsanitize=undefined etc. to gcc execution (for standalone toolchain uses), or
specify NDK_DEBUG and make sure to kill cmd-strip for ndk-build

After those struggles, I could finally get lldb working with my libfluidsynth.so. It was a long journey.

I ended up to write a lengthy article about these kind of tricks for our indie tech book for that TechBookFest5 (in Japanese) from this experience...

Fighting against SIGILL

The SIGILL issue I mentioned at the top of the previous section actually took a while... the debugger did not give much information about "why" (while it gives "where" which is still informative). The relevant function is:

static int chunkid(unsigned int id)
{
    unsigned int i;
    const unsigned int *p;

    p = (const unsigned int *)&idlist;

    for(i = 0; i < sizeof(idlist) / sizeof(int); i++, p += 1)
    {
        if(*p == id)
        {
            return (i + 1);
        }
    }

    return UNKN_ID;
}

It was weird especially because the problematic code was working fine in the past.

Actually, it was problematic. This is what clang reports:

/sources/fluidsynth-midi-service-j/external/fluidsynth/src/sfloader/fluid_sffile.c:494:9: warning: 
      cast from 'const char (*)[117]' to 'const unsigned int *' increases
      required alignment from 1 to 4 [-Wcast-align]
    p = (const unsigned int *)&idlist;
        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 warning generated.

If you google the part of the message "increases required alignment from " then you'll find that it could indeed result in undefined behavior such as SIGILL.

An unsophisticated workaround fixed the issue.

It is possible that the latest NDK changes from gcc to clang or possible compiler option changes in Fluidsynth build (in CMakeLists.txt) triggered the breakage, but I have no precise idea.

Fixing audio glitches

Even at the previous working state with old Fluidsynth port, the audio playback was "glitchy". There was always noise inbetween, which was like it had some blank between synthesized samples.

There was also annoying warnings that OpenSLES spewed onto device logs saying that there was insufficient playback buffer, meaning that there were too much synthesized buffers to enque, before consuming them. It looked like I was enqueuing too much.

Before going forward, I should explain a bit about the buffering internals. There are two approaches to enqueue synthesized audio buffers:

Run an isolated audio loop. Get synthesized buffers and enqueue them while it's alive.
Register OpenSLES callback. Get synthesized buffers and enqueue them within the callback.

In the earlier codebase, I had a hacky workaround to "adjust" the latency to calculate exact buffering time, which often adjusts timing with synchronous usleep() calls. It still made sense when it's running an isolated audio run thread (the former model), but only if the buffering was at "earlier than the expected schedule". And it didn't make sense to try to adjust latency along with OpenSLES callback.

And to make investigation complicated, it took a while for me to find that the buffering option switching by settings API didn't work as expected (due to bogus default value retrieval in my own code). I was also confused by two different "callbacks", one for OpenSLES and the other for Fluidsynth. After all, precise understanding of code and code cleanup to reduce confusion led me to the right solution...

After fixing this, all those buffering related issues are gone and the port became really usable. It still needs some settings to get working in good state, but I'm quite happy with the outcome.

Next step

Android audio situation keeps moving forward. Google had introduced new AAudio API since Android O (API Level 26) which brings chance for lower latency by handful of means (direct buffers, high priority audio callback, low latency mode specification in the API). Google first stated that AAudio will be backported to earlier Androids, but what realized instead was a new audio API called Oboe that provides an unified API which switches two backend implementations (AAudio and OpenSLES).

Oboe was at preview state when I wrote the previous post (or, it didn't even exist when I started porting) but now that Oboe is officially stable, it makes more sense to support it instead of OpenSLES. It is even fair to say that supporting OpenSLES became totally redundant... therefore, the next step is to support Oboe.

managed-midi updates

Atsushi Eno — Thu, 31 May 2018 03:16:46 +0000

Back in January, I wrote an introduction blog post about managed-midi. It's been 4 months since then, and there are couple of updates, so it's nice to sum them up.

MIDI input supported

I have a few MIDI input devices now, especially Seaboard Block is cool (it's quite useless for my actual use though :p).

Maybe you think I'm just showing off my cool toy, but it's slightly more than that. Seaboard Block is based on BLE MIDI, which used to be supported only on iOS. Android 6.0 supported BLE MIDI as part of its Android MIDI API too. But Linux had no chance to use it until fairly recent Bluez and Linux kernel got support for that. I'm on Ubuntu 17.10, so I also needed to switch to some patched build.

Anyhow, I got some reason to work on MIDI input support in managed-midi and it's done there. I also have M-AUDIO KeyStation Mini 32, but those two devices give totally different input messages... what Seaboard brings in is MPE - for each keypress it sends pitchbend, PAf etc. accordingly to the specification, which is a lot of information (like accelerometer multi-axes inputs). It will be fun once I can think of apps and support nicer handling of those MPE messages.

netstandard2 based NuGet packaging

I don't care much about nuget packaging, but it is of some goodness to spread use of this library (which, well, again, I don't care...). As explained in the previous post, Xamarin.Mac Full project is blocked and it's unchanged. Probably few people cares, but I do care - Xamarin.Mac full is the only profile that I can get Xwt working, and therefore my xwt-based xmdsp working) too.

Anyhow Xamarin, which brings the primary reason for me to "package" managed-midi, has migrated to .NET Standard 2.0 world from PCL-based world. So I moved forward too.

There should be .NET Core version of this library too (unlike the netstandard2.0 library, it will contain ALSA support etc.), hopefully in the near future.

Precise time controller

managed-midi used to be a dumb, simple MIDI player that never dealt with latency. It was somehow significant and not ignorable, and it was reported a while ago. It's been a big concern for me too.

managed-midi comes with a timer class called MidiTimeManager. Usually we use the default timer, which waits for the specified time span (simple as Task.Delay()), while we can alternatively use virtual time manager. It is just like TestScheduler in Rx.NET. You don't want to wait forever when running unit tests.

So, technically timers can do more than simple and stupid wait. And any MIDI libraries would be dealing with that. I was lazy to actually implement it until very recently.

While it is still impossible to test this improvements (unless I actually play some songs and measure the actual play time within the unit tests...), I use my xmdsp MIDI player to examine the latest accuracy. I used to author quite complicated song files (either copies or original songs) that I use for dogfooding.

What's next

I have no idea TBH. My target is shifting to either raw audio based stuff (such as android-fluidsynth) and some MIDI composition environment. But since managed-midi is foundation for all my existing apps, it will keep going.

TechBookFest: the most passionate book authors sell their indie tech books in Japan

Atsushi Eno — Sat, 21 Apr 2018 13:08:27 +0000

What is it?

One of the big events I have been involved in Japan is "tech book festival" (技術書典). If you have ever heard of Comic Market in Japan, you'd find it similar but the targets are limited to technology.

It is one-day festival in Tokyo, held at Akiba, where all the geeky things get together. The visitors are mostly engineers. Last time it was in October, unfortunately on a Typhoon day, yet there were more than 3000 people. The next one will be on Apr. 22nd (tomorrow, at the time of writing).

Google images about it and see what it is like.

In Japan, and I assume everywhere in the world, book publishers are shrinking and the industry is sunsetting. Programming and engineering books are not exceptional. But on the contrary, this festival is super active. How is it possible??

There are some factors that make it happen.

There are lots of "circles" (independent people or communities who write books), and they don't have to be very popular. The first TechBookFest was done with 50-ish circles, and the next one will be 246 of those. Their topics are often quite niche, sometimes super niche. They write about whatever they are most interested and would like to write. Just like comic authors at the Comic Market, they don't care much about sales - or they do care, but it's not the top priority. They are not publisher companies and it's not commercial driven.

And since it's a face-to-face festival, it is not only about selling and buying books. They can have some (brief) talk about the technology the circles are writing, and get connected if they want. Similarly, we, the event organizers, lay out those similar circles close, so those neighbors can friend and chat. I have my own circle about Xamarin too, and it is laid out right next to the circle for React Native next time. Flutter circle is on the counterpart too. (Actually we are already all friends.)

Indie books get popular

So you think they won't sell at all? But there are 3000-ish visitors who are really interested in technology. What happens if there are such a lot of visitors browsing all the books at the market? Tenth of people, often hundreds, get interested in your books. It does not have to be sold for thousands. And the number is power. You are not alone to get attraction. There are >200 similar authors.

One of an interesting move that happened from the festival is that there are commercial book publishers (they can also join, sell and exhibit their publications at the festival) that are always looking for interesting books that are in commercial quality.

For example, at the second fest we have published two books about Xamarin (I tried to organize all into one book, but the authors were too passionate and wrote a lot of pages so that I had to split them).

I made a minimalist promotion page for that (which is written in 199Xs markup technology), but it didn't matter. Most of the promotions were done by the co-authors, and those 100 copies for each book were all sold out in 2.5 hours.

I have to say, our book was supreme as of the time of writing, written by top local developers, so some of the publishers got contacted with me if we were interested in publishing those books commercially. So we did.

The next book at the next festival (TBF3) didn't sell that much, but it is still commercially published too. I'm not interested in managing everything for commercial publishing, so our next book won't go commercial (I decided so). But I'm totally happy with indie publication. The commercial publisher takes 85% of the revenue for printed books, which is, well, absurd. We can publish by our own and what they do is only to make some edits and manage everything for Amazon/Kindle (I can do that too).

That's a "successful" case, but you don't even have to be "successful". It's not really the point when you are enjoying the festival. Your publications don't have to be "great". You can find inspiring books, even better if you are distributing your books of your own best interest, and find friends of the same interest.

For the upcoming one, I wrote a book by myself about my Music Macro Language (MML) to MIDI compiler, explaining how to use it, what the syntax is like, and tips for authoring each instrument part e.g. for bass, guitar, drums and keyboards. It is kind of normal content for MML authors, but MML itself is super niche, technology from 20th century. But after I announced it, I received a handful of (not many) requests for and interest in that book ("I want your MML book!"). They are "music to my ears" ...!

The event organizers

The entire festival is managed by two organizations: TechBooster is a software engineer community. The group has been publishing their indie books about Android/mobile and Web at the Comic Market, and now the festival by themselves too. tatsu-zine.com (達人出版会) is an ebook publisher that has been for many years. I'm one of the members of the former and sometimes write about Android stuff.

After some experiences at the Comic Market, the leader started saying like "what if we organize our own festival for tech books?" ...That sounded crazy (the leader always have some crazy ideas), but it somehow realized, and now it's huge.

The core event organizer group is of a few people, and there are like ~50 staffs on the festival day. The group prepares all the websites, organizes the list of circles which visitors can bookmark them (and those at circles can see how many bookmarks they get), support them by several means, even offering online payment system will mobile apps. It's super busy activity especially in the week before the day.

They are highly skilled developers and they often solves the problems technologically. For example, at TBF2 we had to make a lengthy line for waiting people to enter. At TBF3 we have a ticketing system and web page which notifies visitors the maximum ID number of the ticket ("People with ticket number XXX can enter") and the stressful queue is gone.

(And of course the organizer asks for support in money too - each circle has to pay 7000JPY, but that's of course not enough to run the entire event. That's mostly the leader's job.)

How we compose books

TechBooster takes an interesting approach to compose books. When we are authoring them, our circle uses Github repos (in my private account) that are tailored for it. We use a specific markup language called Re:VIEW as the text format, which is suited for book rendering to PDF (using latex) as well as HTML and EPUB. The group publishes an open repository for ReVIEW template as well as their open book on How to write a book using Re:VIEW. There is even a Docker image for Re:VIEW.

On the editor side, we also have editor plugins support for Re:VIEW too. Atom has the first support, and I brought it to VSCode too. It supports preview, spell-check-like editorial helpers, and jump to sections.

Those drafts written in that simple markup language is easy to manage on Github - easy to get diffs, comment, create pull requests, integrating CI builds (for books!) and so on. It's very modern way. It's possible because almost all people in the group are software engineers and know how to git (we often have designers too).

The approach is then taken by many circles and thus helped growing the tech book community. Now I have my own circle for Xamarin too, and it's also based on their project template.

After writing the first draft, we have some "review" process. Usually TechBooster does mutual review each other, but this time for my own circle there were not many authors, so I only asked some friends to do it for our drafts. They did awesome job, so I could make a lot of improvements.

How they go paper

I usually ask my friend painter to draw something I indicate (I pay for it), but this time (for the upcoming festival) I had to do it by myself, and it was hackily done.

There are many small printing companies which offers support for indie books (同人誌 in Japanese) and printings can be done by sending a book body PDF and a cover page PSD like this:

Those small companies are used to this kind of books because of the huge printing need at the Comic Market. There are similar domain-specific events (which we often call "[xyz-]only event" which deals only with [xyz]-stuff) and we TechBookFest are just one of those.

I get familiar with command line tools for manipulating PDFs like pdftk or ImageMagick, as well as dealing with coverpages templates using Krita (which is AFAIK the only tool that I can generate Photoshop *.psd files in CMYK on Linux desktop).

Some of the printing companies are "supported" by the event organizer, and they ship the printed books to the festival venue (for free for my case). The organizer group takes a lot of care about the circles' activities.

I'm writing this because...

Did you get interested? Basically it began with a small independent activity, grew up in public manner. How about your city? It was possible in Tokyo. There weren't ComicCon before but now there are. How about book festival?

Fluidsynth for Android

Atsushi Eno — Sat, 10 Mar 2018 08:59:21 +0000

Motivation

One of my unknown projects is Fluidsynth for Android. Since Android 6.0, it supports its own MIDI API as well as MIDI connection over BLE, so it became possible to connect MIDI devices to Android (if you would like to read my Japanese article, it's part of the book "Android Masters!"). A technically-looking-cool feature is that it supports virtual MIDI device services so that anyone can implement a MIDI device service that can provide either MIDI input devices or MIDI output devices (or both) that other apps can connect as clients and play them just like other operating systems (Windows, Mac, Linux, iOS...).

It was designed to make it capable to run a software MIDI synthesizer through the service, so why not porting any of the existing bits? I thought so, and ended up to bring Fluidsynth into Android land.

Build system

It was not simple; first of all, it does not make sense if there is no sound output. Fluidsynth is a software synthesizer that supports various audio APIs but not for Android. For Android, AudioTrack and OpenSL ES are the available choices (when I was implementing it; there was no AAudio nor Oboe). Fluidsynth has its audio abstraction layer in their drivers source directory, so the only thing I had to do would be just to add another one there, right?

It was not that simple.

First, fluidsynth needed to be built with Android NDK toolchains. The original fluidsynth does not support Android builds. It would be particularly because of glib dependency; there is no intuitive way to build it for Android. glib is autotools-based and NDK did not play with it nicely.

... Wait. There are handful of glib-dependent apps and libraries that are known to work on Android. There is Gimp, right? ... but it runs on top of some weird X server. Next, how about GStreamer? It is known to support Android. How is it built? ... that's how I found Cerbero build system. It builds everything, including glib, in autotools manner, for Android as well as other targets.

All those dependency libraries can be built with "recipe", which customize each library build. cerbero's ultimate purpose would be to build GStreamer, but it can be anything if a recipe is added to the tree. And it was quite easy to add fluidsynth to the catalog.

I had to make some changes to support custom Android NDK setup, so my cerbero tree ended up to become an incompatible fork with the original tree, but I could build libfluidsynth.so for Android in the end.

Everything ended up to become this repo: https://github.com/atsushieno/android-fluidsynth/

Android/OpenSLES implementation

Second, I had to add opensles implementation. The source structure was nice enough and I could easily add fluid_opensles.c to the tree. A minor problem was that there was almost only one reference sample by Victor Lazzarini (his blog used to be publicly visible but now it's private...). Even samples from an NDK book published in Japanese were almost the same as these samples.

One another thing I had to implement was to support custom stream loader for SoundFont files. Fluidsynth only offered filename-based loader which simply used local file I/O. So I had to extend fluidsynth itself to accept custom SF loader - the abstraction API and Android assets-based implementation.

Since my application is written in C#, I added those extensions to my P/Invoke wrappers (that's only what I had to do - if I were using Java, I'd also have to add JNI entrypoints too...).

NFluidsynth and FluidsynthMidiService

Fluidsynth is cross-platform and works fine on Linux, which makes it easier for me to develop C# binding. Fluidsynth itself works as a virtual MIDI synthesizer, but to bridge from the system's MIDI service entrypoints, we have to make fluidsynth functions callable and map from those entrypoints.

Therefore I made binding for fluidsynth API using P/Invoke, released as https://github.com/atsushieno/nfluidsynth . And making it as a Xamarin.Android library is almost at zero cost. I only had to care about Android-specific extensions built only for Android.

At last, I created an Android MIDI device service on top of this NFluidsynth.Android. To build such a service, we have to implement android.media.midi.MidiDeviceService. The entire API is super weird regarding inputs and outputs directions, but is not very difficult to implement.

During this attempts, I found that supporting Android MIDI API is almost no benefits. My managed-midi API is cross-platform, inspired by Web MIDI API, and it makes more sense to rather implement Android backend for it. Therefore I came up with two implementations; one for MidiDeviceService and another for my API.

They end up to become this repo: https://github.com/atsushieno/fluidsynth-midi-service

CMake switches and android-fluidsynth port

All those works were done in earlier years, so it is kind of weird that I write this post in 2018. I thought there would have been more software MIDI synthesizers for Android being released, but it did not seem to happen. What I got instead was, a handful of inquiries about my android-fluidsynth port. Since the entire build system is quite tricky, most of those who attempted to build failed hard (I feel sorry for that...).

One of the reasons it is kept undocumented was that the current state is (to me) very temporary - when I started this project, it was hosted at sourceforge and it was based on autotools. Now it is hosted at Github and the build system moved to CMake.

CMake is problematic right now - cerbero technically supports CMake, but it never seemed to support Android. What cerbero does there is to specify custom toolchains (CC, LD, etc.). And when you override some toolchains specified by cmake toolchains config (there is one in Android NDK) ... it will detect inconsistent build specification and restarts the build again, without specified cmake options(!). That ends up to ignore my build options and the build fails. I still find no answer to solve this problem.

Graduating Cerbero is an option, and the option comes with a handful of choices - cerbero is in our build system because it resolves glib dependency problem. To eliminate cerbero, we have to find ways to resolve glib build issue.

VolcanoMobile/fluidsynth-android is an option; it removed all glib dependencies. This tree however became incompatible with the original tree, and I don't feel comfortable with that. However, there is an effort to incorporate my OpenSLES implementation into that diverged version. I'm thinking to switch the basis for binary build artifacts to the ones from this tree.

Other choices are Google cdep (I don't like their basic idea of forking sources to make their own changes, which makes it merging origin unnecessarily messy), or incorporating mono eglib instead of depending on glib. But so far I'm okay with the glib-less port above.

UPDATE (2018-03/12): eglib doesn't provide gthreads, so it's not an option.

What's different from user's point of view

Fluidsynth automatically chooses the available options to build, for each platform. And opensles will be the default driver in my Android port.

There are some changes in SoundFont loader API and Android developers would like to use it (explained a bit, above). But other than that, there is no difference from the original source tree.

managed-midi: the truly cross platform .NET MIDI API

Atsushi Eno — Wed, 10 Jan 2018 14:54:42 +0000

I have a (surprisingly) long term project called "managed-midi" which is to offer cross platform C#/.NET MIDI access API. My "cross platform" here is not a marketing term; it targets Linux (ALSA), macOS (CoreMIDI) and Windows (WinMM), along with Android, iOS and UWP.

Actually, my managed-midi project is not about cross-platform goodness. It's more of a structured music composition as well as messaging foundation. This is my primary motivation to develop this library.

But I'm going to discuss cross-platform part.

Wasn't there any existing effort to provide Midi access API? The only project I can think of is NAudio, which only cares about Windows. My primary desktop is GNOME on Linux, so it's no-go. This is a typical huge problem in C# developer ecosystem that they only care about Windows.

Audio and music libraries are always like that, and few people have interest in cross-platform AND platform specific development. This development situation is similar to developing "bait-and-switch" PCLs, but like I generally don't care about Mac and UWP, people don't care about platforms they don't use.

It is kind of ironic that MIDI features are categorized within very platform specific API groups, whereas MIDI itself is designed to be device independent.

The situation in C++ world is however changing. VST is now available on Linux too (I mean, the original library from Steinberg). We are seeing new-generation DAWs working on Linux (Renoise, Bitwig Studio, Tracktion WaveForm etc.).

wrapping around cross-platform MIDI API

Back in 2009 when I started launching this project among other small projects I had in mind (when this project didn't even have a name and repo), I found PortMIDI, which is a cross-platform MIDI library that supports Windows, Mac and Linux, and I found it cool. I didn't want to deal with platform-specific MIDI APIs, especially whatever I was not familiar with. Instead, I wrote a P/Invoke wrapper and OO-style API around it. I think my idea was good (I still kind of) - it is important to quickly get what we need, right? Those smart native developers dealt with the platform specifics, and I made use of the outcome.

That was the beginning of this library. I just wanted an SMF parser and a MIDI player that plays MIDI songs that I can compose using my music macro language compiler mugene. There wasn't even appropriate abstraction.

It was however somewhat messy to get apps that are based on this library working. My primary .NET runtime has been always Mono, and I was primarily on Windows at that time. I couldn't care much about Linux. Since portmidi is a pure third-pary library it always required binary shared libraries. portmidi was painful to me on Windows since it required JNI header files to build. And it has to be offered in 32bit for Mono (which supported only 32-bit x86 on Windows at that time) and 32bit binary didn't work on .NET Framework for some reason. I was not familiar with them. I almost dumped the idea of offering binaries by myself.

A few years later, I found RtMidi and it was simpler. Unfortunately it was C++ only, so I wrote C wrapper for that (which I ended up to contribute to the project). It had the same issue as portmidi too but didn't require JNI stuff, so building binaries was not so painful. Moreover, the development was active (at that moment).

There was still no proper abstraction, but it was implicitly done. The abstraction was done only as MidiPlayer class (then I had PortMidiPlayer and RtMidiPlayer). I didn't care much about that.

MIDI API abstraction

The primary reason I was developing managed-midi was to write usable MIDI application. So it does not have to be C#. There has been another effort to bring MIDI to cross-platform world - Web MIDI API. It is expected to be working on Web browsers, but right now it is only Chrome which supports it.

Web MIDI support in early days was not actually great for Linux software synthesizers (Chrome supported only Mac at first, and software synthesizers need special permission to be enabled), but its API structure was informative. It gave me the basic ideas on what we can/should provide in any platform and what not.

Therefore, I started refactoring the entire API structure. What's good with Web MIDI API was that it is kind of OO-style and closer to C#, compared to former C libraries.

Thus, current API structure is like:

IMidiAccess
IMidiPortDetails
IMidiPort
- IMidiInput
- IMidiOutput

The MIDI messages are represented just as byte arrays. It is same as Web MIDI and RtMidi (portmidi was different, had its own message type).

Implementing direct access to native API

Even with RtMidi, I was reluctant to build those libraries for Windows (I already switched my primary desktop to Linux) and Mac (I hate Apple's attitude against Adobe Flash, dumped my iPhone and switched to Android because of that), especially since I had to provide my custom library that included C API.

After abstracting the API, I started to think that I could implement platform-specific implementation and switch the default implementation per platform so that I didn't have to bother to build native binaries anymore.

Though the first problem I was faced was no P/Invoke availability in PCLs. I quickly thought that it's not suitable for PCLs. But there is another tech trend in .NET: PCL with "bait and switch" technology that makes it possible to provide a common API with platform-specific implementation. Technically this problem seemed resolved.

The first native implementation target was Windows. Windows at this state means: Windows desktop and UWP (I don't care about WinRT, there is no official MIDI API and those users are used to be limited their ability). As of that time, what I wanted was rather getting my visual MIDI player xmdsp seamlessly working on Windows and Mac as well as Linux, and it was on top of xwt which is based on WPF and thus desktop .NET Framework. Therefore WinMM matters. WinMM was easy. UWP is more organized API and wrapping around it was easy, but since I don't have any managed-midi apps it is totally untested.

CoreMidi was different. It did not even start with desktop. What I actually thought was to get managed-midi working on Android (and it was not even for Android MIDI API - it was for my fluidsynth-midi-service project, but it's going to be too long to discuss that here). Anyhow mobiles got in sight at that time. Android MIDI API is shitty with the class names, but wrapping around it was easy.

While I wrote CoreMidi implementation for iOS, I didn't have any managed-midi apps that work on iOS, so I didn't even try it. Even on Mac, I tried only once when Xamarin.Mac started working fine with Xwt with CoreMidi (there was complicated library resolution bug that I had to wait for fixes in 2017). API wise, CoreMidi was a bit more complicated than others due to their concept of endpoints.

And... finally ALSA. I didn't want to work on ALSA because rtmidi just works, building rtmidi was always easy, and ALSA is complicated. But copying librtmidi.so everywhere is messy (like this issue) and as long as I am dependent on rtmidi I could not resolve some weird issues like this, so I ended up to learn about ALSA and implemented it as alsa-sharp.

Fundamental problem with the "bait and switch" trick in .NET Standard

At this state, managed-midi lacks a lot of features like input support everywhere (surprisingly? I didn't need it yet) and device connection state detector (which is part of Web MIDI API so it should be implemented too). I only cared about output (my primary goal is to get a MIDI player). Device detection is complicated on ALSA as it has to be done outside ALSA, and probably WinMM which doesn't provide the feature too.

NuGet packaging is another bit problem - while I intended to build a cross-platform library, it does not take shape of "bait-and-switch" PCL or .NET Standard library. Here is why: Xamarin.Mac full profile requires its own ProjectTypeGuids, which means it is basically different from .NET Framework desktop profile. They have to be different. And I cannot depend on Xamarin.Mac when I'm working on this library because I'm not on Mac (even if I had Mac, no one should be required to work on Mac anyways) and the entire desktop library cannot be Xamarin.Mac specific (see this post for the PCL/NuGet target moniker).

Therefore managed-midi has two different assemblies for the same netstandard moniker (net461). That makes it impossible to package one of either. Since there is no reason to prefer Xamarin.Mac specific assembly for net461, it is basically ignored.

This is something .NET Standard designers should resolve.

First post to dev.to

Atsushi Eno — Wed, 10 Jan 2018 14:54:24 +0000

I haven't blogged in English for a while (I do in Japanese) and that's mostly because I was always wondering where to post. dev.to looks good enough to me. My posts can be very short or very long, up to my motivation and feelings.