DEV Community: Mariano Martinez Peck

Segfaults: GDB Scripts to the Rescue

Mariano Martinez Peck — Tue, 10 Nov 2020 11:53:36 +0000

We recently discovered that we would very sporadically get a segmentation fault when running the whole test suite of the VAST Platform (VA Smalltalk) on Linux.

During our initial investigation, we realized this was not something new, but that the problem had existed for many years. It happens only on Linux (not Windows), and with all program variants: with and without JIT compiler, with and without memory sharing, 32 and 64-bit, etc. At least we knew it was not related to those broader differences.

In the end, we were able to find and fix the bug. In this post, I’ll outline some of the experiments we performed and conclude with the final GDB approach that ultimately did the trick.

Attempts at running the whole test suite

For the internal development of the VAST Platform, we use a huge test suite of about 16,000 total tests. When diagnosing an issue like this, the first obvious step was to just run the suite and see if we would get the “segmentation fault”. Obviously we did not. Otherwise, we would have fixed this long ago. We then tried randomizing the order of the tests, running the suite multiple times, etc. Still no luck.

One of the peculiarities about this issue is that there are very few scenarios that could cause a segmentation fault in VAST. But when it does occur, it normally dumps a vmtrap.log which provides you a lot of useful information for debugging. Unfortunately, in this case, the log was not being produced. In addition, when getting this segmentation fault, it seemed that VAST was doing a regular successful exit (quitting the image, and shutting down the VM).

That made us think that the problem could be related to the “exit code” in the Virtual Machine. The next step was to write a bash script that would do a lot of image starts & exits so we could observe the exit codes:


#!/bin/bash

# "Settings"
vaRoot="/usr/local/VASmalltalk/10.0.0x64"
vaVMPath="$vaRoot/bin/es"
workingDirectoryPath="$PWD"
log="$workingDirectoryPath/ctest.log"

# copied them from environments directory.
# The icx needs to have SUnitOneClick map loaded
imagePath="$workingDirectoryPath/abt.icx"
iniPath="$workingDirectoryPath/abt.ini"

cnfPath="$vaRoot/abt.cnf"
# The -no_break is strictly necessary because else it looks like
# the VM randomly freezes and we get defunct processes ..
# so the only workaround for now is to disable the break process.
vaVMArguments=" -no_break -mcd -msd -i$imagePath -ini:$iniPath"
# This testcases.txt file can be autogenerated with the Smalltalk code:
# CtestSunitRunner basicNew createTestCaseList
testCaseFile="$workingDirectoryPath/testcases.txt"
echo "Testcase file: $testCaseFile"

# Doing a cd is important here because else the ICs won't be found in the
# defined IC_ROOT in the .ini
cd $vaRoot
# If we don't specify this LANG or any other compatible one
# we would get quite some test failures
export LANG=en_US.iso88591
# Necessary, else the esvm40.so for example is not found
export LD_LIBRARY_PATH="$vaRoot/bin"
# Just to always start with a fresh log
[-e "$log"] && rm "$log"

echo " ======== Starting process ======="

set -e
while IFS='' read -r line || [[-n "$line"]]; do
    # Print some status on the console...the rest..directly into the log.
    echo "Line: $line"
    # Below two lines create forces a new abt.cnf files and appends
    # the necessary code to run the test with Ctest
    echo "PostStartUp ! " > $cnfPath
    # This line just extracts the class name from the line and appends the
    # code to abt.cnf
    echo "SunitRunner run: #`echo "$line" | cut -d '#' -f2- | cut -d '-' -f1` !" >> $cnfPath

    # Start VA VM with all the necessary arguments and redirect stdout/stderr
    # to the specified log file
    $vaVMPath $vaVMArguments < /dev/null >> "$log" 2>&1

    echo "Line finished"

done < "$testCaseFile"

echo " ======== Process finished ======="
echo `cat "$log" | grep "CTestFailure"`

I don’t want to get into the gory details as that’s not the focus of this post, but basically, the script does the following:

Reads the file testcases.txt which has all TestCase subclasses (one test class name per line) of the whole test suite.
For each test class, it auto-generates a file called abt.cnf that is automatically read by VAST at startup, and that file includes the code to run that test (SunitRunner run: #MyTestCaseSubclassExample) .
For each test class, it launches VAST. At startup, it runs the test. When done, it will print the test results and exit.
It will keep repeating steps 2 and 3 until the end of testcases.txt

Finally, when running this bash script, we were able to sporadically reproduce that segmentation fault. (Sometimes we needed to run this script a few times to get to the crash.)

Trying to run with GDB

Ok…we were able to somehow, every once in a while, reproduce the crash with the “release” VM. The next step was to compile the VAST VM in “debug” mode and cross our fingers that we’d be able to still reproduce the bug. (Sometimes life isn’t easy and bugs don’t show up with the debug flavor).

I normally compile and run the VAST VM under GDB with a command like this:

export VM_COMPILED_BIN=/home/mpeck/Instantiations/git/vm-devel/devel/build/bin; \
export VA_ROOT=/usr/local/VASmalltalk/10.0.0x64; \
cd $VA_ROOT ; \
export LANG=en_US.iso88591 ; \
export LD_LIBRARY_PATH=$VM_COMPILED_BIN:$VA_ROOT/bin; \
gdb -ex run --args \
$VM_COMPILED_BIN/es -no_break -ini:/home/mpeck/Instantiations/SUnitOneClick/abt.ini -i/home/mpeck/Instantiations/SUnitOneClick/abt.icx |& tee gdb.log

Therefore, I altered above bash script and tried to replace the line:


$vaVMPath $vaVMArguments < /dev/null >> "$log" 2>&1

with something like:


  gdb -ex run --args $vaVMPath $vaVMArguments >> "$log" 2>&1

Now, after running that new version of the bash script we received both good and bad news. The good news was that the bug still shows up with the debug VM. The bad news was that gdb ends the session if it’s run in a bash script. Therefore, I wasn’t able to get into the GDB console once the segmentation fault was being triggered.

Narrowing down the reproducible case

Reproducing the crash with the bash script fueled even more of our suspicions relating to the “exit code” in the VM. We thought that maybe it wasn’t necessary to run the whole suite of tests, but rather, just running one single test class hundreds of times could still reproduce it.

In other words, altering testcases.txt with a single line with a single test (TestProcessPrims in our case) and adding an outer loop in bash to repeat the process 1000 times.

Our hunch was right and it still reproduced the problem. Now this was going to be much easier.

Run a program 1000 times with GDB

Finally…we are getting close to the most helpful piece of this post.

We knew that all we needed to do was to run VAST 1000 times with gdb with an abt.cnf like below, but without ending the GDB session when the segmentation fault was hit.


PostStartUp ! 
SunitRunner run: #TestProcessPrims !

Researching online how to do this, we arrived at this useful post that shows how you can write a GDB script for that.

We created the file /tmp/script_file.txt as follow:


set pagination off
handle SIGUSR1 noprint nostop
handle SIGUSR2 noprint nostop
set $n = 1000
while $n-- > 0
  printf "\n\n=========================\nstarting program: %d\n", 1000-$n
  run
  if $_siginfo
    printf "Received signal %d, stopping\n", $_siginfo.si_signo
    loop_break
  else
    printf "program exited\n"
  end
end

And then ran GDB and VAST like this:


export VM_COMPILED_BIN=/home/mpeck/Instantiations/git/vm-devel/devel/build/bin; \
export VA_ROOT=/usr/local/VASmalltalk/10.0.0x64; \
cd $VA_ROOT ; \
export LANG=en_US.iso88591 ; \
export LD_LIBRARY_PATH=$VM_COMPILED_BIN:$VA_ROOT/bin; \
gdb -x /tmp/script_file.txt --args \
$VM_COMPILED_BIN/es -no_break -ini:/home/mpeck/Instantiations/SUnitOneClick/abt.ini -i/home/mpeck/Instantiations/SUnitOneClick/abt.icx

Note the gdb -x /tmp/script_file.txt.

Finally!!!! When running that, we were able to hit the segmentation fault, get into the GDB console, and debug.

The line 96 of estimer.c was what we needed to understand the problem and fix it.

This was an ancient bug. VAST’s main thread sets the ‘done’ flag to be later read by the ‘delay’ thread (estimer.c) and then continues to execute. Sometimes, a rare condition appears when the main thread is done shutting down and cleans the ‘platform globals’ struct where the ‘done’ flag is located. Thus, by the time the ‘delay’ thread attempts to read the ‘done’ flag, it’s actually reading already freed memory causing a segfault. The easy fix is to wait for the ‘delay’ thread to finish after ‘done’ flag is set, but there are of course other possible fixes too. In any case, the bug and fix itself was not the purpose of this blog post, but rather to show how we approached the problem and some tips with GDB.

Conclusion

Reproducing bugs and debugging them is not usually as easy as one would like. In this particular case, it was a learning experience for me how to run a program multiple times from within GDB.

“DLL Hell”: Tips & Tricks to Avoid it in VAST

Mariano Martinez Peck — Wed, 21 Oct 2020 13:50:22 +0000

Delegating tasks from high-level languages like the VAST Platform (VA Smalltalk) to languages like C, C++, Rust via some kind of FFI (Foreign Function Interface) is becoming more and more common.

Ideally, you would like to have everything implemented in your preferred high-level language, but I believe in using the appropriate tool for each problem. Sometimes you need more performance, or sometimes you want to save yourself the costs of implementing and maintaining a library that’s already implemented in another language. The reasons could be many, but in this case, you will end up making a thin FFI binding (a.k.a wrapper) against that library.

Then, when you are in your high-level language and you want to call, for example, a function from a C library compiled into a DLL, you need to search for that DLL (Dynamic-link library) and load it into your applications process space.

That may sound like an easy step, but it can be one of those situations where developers spend a lot of time trying to figure out what is wrong: why a DLL couldn’t be loaded, why another DLL (with a different path) was loaded instead of the expected one, and so forth. This is a frequent problem that is widely known as “DLL Hell“.

In this post I’ll show some tips, tricks, and scripts to help you get out of this situation when loading DLLs from the VAST Platform.

Troubleshooting VAST .ini specifications

In VAST, most of the time, the library filename of the different wrappers are specified in the main .ini file under the section [PlatformLibrary Name Mappings].

For example, below is an extract of it:


[PlatformLibrary Name Mappings]
; Almost all .DLL files referenced by VA Smalltalk have aliases (logical names) that are mapped here.
;
; The keywords (logical names) are case sensitive; the values (dll names) are not.
Abt_Primitives=
AbtNativeQ=abtvof40
AbtNativeV=abtvmf40
BROTLI_LIB=esbrotli40
CgImageSupport=escgi40
CRYPTO_LIB=libcrypto

That way you customize the “physical name” (ex. libcrypto) of a given “logical name” (ex. CRYPTO_LIB). You can use different names or even full paths…for example:


CRYPTO_LIB=my-special-libcrypto
CRYPTO_LIB=c:\Users\mpeck\Documents\Instantiations\libs\OpenSSL_x64\libcrypto

So…let’s see the typical issues and possible solutions.

Be certain of which .ini file VAST is reading

Before you start digging into why a certain DLL didn’t load properly and you start randomly editing .ini files… just be sure you are editing the correct one.

This may be obvious, but I have seen cases where VAST is started from a chain of .bat files (where it’s hard to get to the exact line that ends up executing VAST) or cases where there are also many custom .ini files (read by end-user applications). In these situations, it’s not always easy to “know” which is the exact .ini file being read by VAST itself at startup to map logical to physical names.

The first solution here is to ask the system which exact .ini file was read at startup:


Transcript 
    cr; 
    show: 'INI File: ';
    show: System primitiveIniFileFullPathAndName asString.

This will only help if you are having the problem on a development image or on a runtime one that you are able to modify and re-package it to print this information.

However, if the problem is in a runtime image you can’t re-package, then it will help to know the exact algorithm the VM does to look for the primary .ini file:


 /**
 * Finds the .INI filename.
 * This is called AFTER EsFindImageFileName().
 * Algorithm:
 * 1. If globalInfo->iniName has a name in it (came from -ini: commandline switch),
 * look for (globalInfo->iniName).ini file, answer TRUE if present; else answer FALSE.
 * 2. Look for (globalInfo->imagePath)(globalInfo->imageName).ini, answer TRUE if present.
 * 3. Look for <exePath><exeName>.ini, answer TRUE if present.
 * 4. Look for es.ini, answer TRUE if present.
 * 5. Answer FALSE.
 */
BOOLEAN EsFindIniFileName(EsGlobalInfo *globalInfo)

This way, you can manually check for the existence of these files step-by-step until you find which one is being used.

BTW: VAST 2021 (v10.x.x) is including this information as part of the default walkback header.

Be sure your .ini edits are being taken into account

Again, this sounds obvious, but I’ve seen end-user applications that override the default VAST startup mechanism, have their own .ini reading, hardcoded settings, just to name a few…

So… once you start modifying the correct .ini file to troubleshoot the problem, be sure that those changes are being taken into account. The below code queries VAST to know exactly what have CRYPTO_LIB been mapped to after startup:


| logicalName aliases sharedLibs lib |
logicalName := 'CRYPTO_LIB'.
aliases := PlatformLibrary classPool at: 'Aliases'.
sharedLibs := PlatformLibrary classPool at: 'SharedLibraries'.
Transcript show: 'Alias Key: ', logicalName, ' Value: ', (aliases at: logicalName); cr.
lib := sharedLibs detect: [:each | each logicalName= logicalName] ifNone: [nil].
Transcript show: 'SharedLibraries LogicalName: ', lib logicalName, ' PhysicalName: ', lib physicalName.

So for example, if I edited my .ini file with this line:


CRYPTO_LIB=c:\Users\mpeck\Documents\Instantiations\libs\OpenSSL_x64\libcrypto

But above code prints libcrypto then it means that for some reason my changes are ignored. So….be sure that above code prints what you specified in the file.

Troubleshooting DLL load failure

Assuming you are sure which main .ini VAST is reading and that your changes would be taken into account, you could now start troubleshooting why a DLL failed to load or why a wrong one has been.

Windows DLL lookup order

The most common problem when loading a DLL is that either Windows couldn’t find it through its lookup mechanism or it did but it’s a different version than the one you would expect (which could be problematic).

If you look at the official documentation about the lookup mechanism you will see that there are many factors that could affect it: registry keys such as KnownDLLs or SafeDllSearchMode, custom load via LOAD_WITH_ALTERED_SEARCH_PATH, etc. However…. most of the time you end up using the “default” behavior.

Frequently, it’s hard to identify the working directory, what/where the VAST executable binary is, what directory is GetSystemDirectory or GetWindowsDirectory, etc.

The below script tries to mimic the Windows lookup order and displays what would be the directory for each step:


| string m p | 
string := String new: 256.

m := OSHmodule getModuleHandle: nil.
m isNull ifFalse: [
  p := String new: 256.
  m getModuleFileName: p cbFileName: p size.
  Transcript show: '1: The directory containing the EXE file: ', p trimNull ; cr.
] ifTrue: [Transcript show: 'Can''t find module!'; cr.].

OSCall new getSystemDirectory: string cbSysPath: 256.
Transcript show: '2: GetSystemDirectory : ', string; cr.

OSCall new getWindowsDirectory: string cbSysPath: 256.
Transcript show: '4: GetWindowsDirectory : ', string; cr.

Transcript show: '5: Current Directory : ', CfsDirectoryDescriptor getcwd; cr.

Transcript show: '6: $Path : ', 'Path' abtScanEnv; cr.

And that script could print something like this:


1: The directory containing the EXE file: Z:\Common\Development\VAST\10.0.0x64-b466\10.0.0x64\abt.exe
2: GetSystemDirectory : C:\WINDOWS\system32
4: GetWindowsDirectory : C:\WINDOWS
5: Current Directory : Z:\Common\Development\Images\10.0.0.x64-b466-dev
6: $Path : Z:\Common\Development\VAST\10.0.0x64-b466\10.0.0x64\;Z:\Common\Development\VAST\10.0.0x64-b466\10.0.0x64;C:\Python27\;C:\Python27\Scripts;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\PuTTY\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Git\cmd;C:\Program Files\LLVM\bin;C:\Program Files\CMake\bin;C:\Users\mpeck\AppData\Local\Programs\Python\Python37\Scripts\;C:\Users\mpeck\AppData\Local\Programs\Python\Python37\;C:\Users\mpeck\AppData\Local\Microsoft\WindowsApps;C:\Program Files\PuTTY;C:\Users\mpeck\AppData\Local\Microsoft\WindowsApps;

Now you know which directories Windows will look for and in which order.

Pro Tip: Are you running VAST 32-bit on Windows 64-bit? If so, bear in mind that the function GetSystemDirectory will answer C:\WINDOWS\system32 even though the real physical directory mapped to it is C:\Windows\SysWOW64\. You can read this post for more details.

Oh, by the way, do you also want to grab any of the involved Windows registry keys that could affect the load order? You can do that from VAST too!

The below example gets the value of SafeDllSearchMode


"Registry Hive: HKEY_LOCAL_MACHINE
Registry Path: \System\CurrentControlSet\Control\Session Manager\

Value Name: SafeDllSearchMode

Value Type: REG_DWORD
Value: 1
"

| subKey buffer key answer valueName |

subKey := ByteArray new: 4.
buffer := String new: 256.
key := 'System\CurrentControlSet\Control\Session Manager'.
valueName := 'SafeDllSearchMode'.

 answer := (OSHkey immediate: PlatformConstants::HkeyLocalMachine)
    regOpenKeyEx: key asParameter
    ulOptions: 0
    samDesired: PlatformConstants::KeyQueryValue
    phkResult: subKey.

 answer = PlatformConstants::ErrorSuccess
  ifTrue: [
    subKey := (OSHkey immediate: subKey abtAsInteger )
        regQueryValueEx: valueName
        lpReserved: 0
        lpType: nil
        lpData: buffer asParameter
        lpcbData: (OSUInt32 new uint32At: 0 put: buffer size; yourself)
asParameter.

  (buffer asByteArray uint64At: 0) inspect.
  ].

What’s the path of my loaded DLL?

Another typical problem when dealing with DLLs is that for certain VAST images or machines it works, but for another it doesn’t. If you didn’t specify a full path for that DLL in the .ini file, then it would be useful to know which exact DLL (with path) is being picked-up by Windows on both cases.

For that, you can use below VAST script:


"Disclaimer: this assumes you already loaded the dll into the running process"
 | m p |
 m := OSHmodule getModuleHandle: 'libeay32.dll' asPSZ.
m isNull ifFalse: [
  p := String new: 256.
  m getModuleFileName: p cbFileName: p size.
  Transcript show: 'Loaded DLL: ', p trimNull.
] ifTrue: [Transcript show: 'Can''t find module!'; cr.]

Which prints something like this:


Loaded DLL: z:\Common\Development\Images\10.0.0.x64-b466-dev\libeay32.dll

Another way to do this (outside of VAST) is using the “Process Explorer” tool as explained in this previous post.

Check DLL dependencies

Many times I think “I swear the DLL is there but VAST fails to load it”. It could be that the DLL you are trying to load is in the correct place and Windows is indeed trying to load it, but Windows may be failing to load it because a dependency of that DLL couldn’t be found.

For example…say we tried to load ssleay32.dll and that failed. As shown in a previous post, we can use the dumpbin tool to identify the dependencies of that DLL and be sure that we are not missing anything:

As you can see, in this example, ssleay32.dll depends on libeay32.dll (as well as others). So… if libeay32.dll or any other of the dependencies cannot be found, the load of the root DLL (ssleay32 in this example) will just fail.

Always check dependencies.

Verify the bitness of your executable and the DLL

A variation from the previous problem: “I swear the DLL is there but VAST fails to load it”. If you use both 32 and 64-bit programs you may eventually face this issue. When you are looking at a DLL and thinking Windows should have been able to load it, but it couldn’t even though the dependencies are fine, the other thing to check is the bitness. VAST 64-bit is only able to load 64-bit DLLs…and same for goes for 32-bit.

For this, I don’t use a VAST script but rather a simple technique using a text editor. Very useful to keep in mind.

Conclusions

Hopefully these tips and tricks are useful for you! Do you have another trick to share? I would love to hear about it.

Troubleshooting Applications Running on Windows

Mariano Martinez Peck — Thu, 08 Oct 2020 19:33:13 +0000

Over the past few months, I’ve been troubleshooting hard problems that have appeared when running the VAST Platform (VA Smalltalk) on Windows. Some of the problems were indeed bugs (like sockets leaking under a particular scenario) and some were just Windows or customer issues.

Regardless of where the problem was, I learned much about certain tools and tricks that could help me again in the future or possibly help others too. Hence, the reason for this post.

Process Explorer

I have heard before about the “Process Explorer” tool, but I had not used it until recently. This is a really small application (3MB) that allows you to analyze many aspects of the running processes: loaded DLLs, threads, opened file handlers, and many more.

In my case, the two features I used the most were the ability to know:

(1) Which exact DLLs (with full path) were loaded in the process (for one of the problems a wrong version of the DLL was being loaded)

(2) The list of opened file handlers (another problem was a leak getting close to the max allowed file handlers per process)

PRO TIP: You don’t even need to install “Process Explorer” (sometimes installing programs on large companies isn’t easy) as it comes in a kind of “portable” executable that you just double click.

Dumpbin

A shared library could (and very likely will) depend on other shared libraries (dependencies).

One typical problem when you dynamically load shared libraries is that the load of the library fails because the operating system can’t find one or more of the required libraries.

In Linux, we would normally use something like ldd mylib.so and check for the not found. But for me, it wasn’t obvious how to do that on Windows. I knew about the “Dependency Walker” but I also understand that it does not support Windows 10 very well, nor is it officially supported by Microsoft.

Finally, I found the “dumpbin” tool which is officially supported by Microsoft and allowed me to do what I needed…and more. If you have Microsoft Visual Studio, then you probably already have the tool (although it’s very very hidden). Otherwise, you can install the “Windows 10 SDK“.

The feature I needed this time was /dependents as you can see in below example:

You can check the documentation for all the possible command line arguments.

The Microsoft Error Lookup Tool

I am a little bit ashamed that I discovered this tool only just recently. A customer reported a Windows error in hexadecimal format. Sure, I can Google that exact hexa error code, but the Microsoft Error Lookup Tool is very handy. You can just pass the hexa as an argument, and I will show all the places Windows found for that error and also print in which C header file it’s defined.

Here is a simple example with the hexa code 0x80070583 I was investigating due to a COM initialization problem:

And here is an example with more than 1 result:

Forcing Windows to re-load manifest files

VAST Platform 9.2 comes with a full native HiDPI support for Windows. For that, it needs some lines in the manifest files:

 <compatibility xmlns="urn:schemas-microsoft-com:compatibility.v1"> 
   <application> 
       <!-- Windows 10 -->
       <supportedOS Id="{8e0f7a12-bfb3-4fe8-b9a5-48fd50a15a9a}"/>
       <!-- Windows 8.1 -->
       <supportedOS Id="{1f676c76-80e1-4239-95bb-83d0f6d0da78}"/>
       <!-- Windows 8 -->
       <supportedOS Id="{4a2f28e3-53b9-4441-ba9c-d69d4a4a6e38}"/>
       <!-- Windows 7 -->
       <supportedOS Id="{35138b9a-5d96-4fbd-8e2d-a2440225f93a}"/>
       <!-- Windows Vista -->
       <supportedOS Id="{e2011457-1546-43c5-a5fe-008deee3d3f0}"/> 
   </application> 
 </compatibility>
 <application> 
   <windowsSettings> 
     <dpiAwareness xmlns="http://schemas.microsoft.com/SMI/2016/WindowsSettings">PerMonitor</dpiAwareness> 
     <dpiAware xmlns="http://schemas.microsoft.com/SMI/2005/WindowsSettings">True/PM</dpiAware>
   </windowsSettings> 
 </application>

One problem we detected is that Windows seems to cache the data of the manifest file even if you edit it and reboot. We researched a lot for possible workarounds and a good one was found in this post. However, we found an easier trick that worked for us. Example:

Wrong Locale when running your program as a Windows Service

If you normally use the “US” region on Windows you may not have faced this issue. If you have custom Locale settings (ex. german-german) you may experience they are not being properly detected by your program when running as a Windows Service, but they are when run from a console.

When you run a program from a normal CMD console, it uses the normal Windows’ user logged in (ex. “Mariano” user). But when you register your application as a service, the user to log on, will probably be “Local System” (the default). You can confirm this by looking at your registered service and the column “Log On As”.

The problem is that the “Local System” settings for Locale are not necessary the same as your current Windows user (ex. “Mariano”). The solution to this problem is to specify the Locale information also to the “Local System” user. Here is how you do it.

OpenSSL for Windows

OpenSSL is becoming more and more critical every day and not only on HTTP but in SMTP, MQ, SFTP, SSH, just to name a few. As you probably know, there is no “official” binary download for Windows. That means that there are a few different 3rd party places from where you can download pre-compiled binaries. You can even compile it yourself. This, combined with the fact mentioned above that OpenSSL is become more and more used, could be nightmare if you are trying to load OpenSSL dynamically and relying on Windows DLL lookup mechanism.

The mentioned Process Explorer may help you realize which exact OpenSSL dll you are loading, but after that, it would be helpful to know more about that dll. Which version was it? Which parts were included when it was compiled? To help resolve those questions, we found this great website: https://www.howsmyssl.com/. You can do a normal HTTP GET with your favorite client (curl, wget, a web browser, etc) and it will answer a nice HTML response with a lot of info. Example:

Conclusion

This will probably just be a reminder for “future me” when I encounter these problems again. But in the meantime, I hope this was helpful for others too.

Thanks for reading.

Deploying VASmalltalk on Amazon AWS ARM servers

Mariano Martinez Peck — Tue, 14 Apr 2020 12:26:22 +0000

A while ago, I did some experiments with Amazon AWS and EC2 instances. Back then, they weren’t very different from a VPS except that you were charged based on the usage and the servers were easier to grow or shrink.

But… much has changed since then and AWS now offers many services and features. One of them particularly caught my attention since it was announced: ARM-based instances.

As you may have read in my previous posts, I have been doing a lot of work with the VA Smalltalk platform together with Docker on ARM, Raspberry Pi and IoT, Machine Learning on the Edge, etc. Therefore, trying an EC2 ARM instance with Docker and running a VA Smalltalk program inside the container was just logical.

In this post, you will read how easy it is to deploy a VA Smalltalk web application into an A1 EC2 instance.

AWS Account Setup

Funnily enough, this is the step that took the most amount of time! Hahaha. If you don’t have an AWS account, you need to set up one with payment information and all the other necessary info.

Once that is done, the next step is to create an ACCESS_KEY and a SECRET_ACCESS_KEY. You can read this guide for details.

Selecting the EC2 Instance

Here you can see the list of all possible EC2 instances. There are both ARM and Intel-based instances (A1, T3, M6g, etc). For each instance there are several possible configurations (numbers of processors, memory, hard disk, GPU, bandwidth, etc) for example: a1.medium, a1.large.

In addition, the type of instances are split in categories such as “General Purpose”, “Accelerated Computing”, etc., so it’s really easy to find an instance and configuration that matches your specific needs and costs.

For this demo, I will use an A1 large, which is a general purpose instance with an AWS Graviton Processor featuring 64-bit ARM Neoverse cores.

Creating the Instance

Once you have selected the type and configuration of instance you want, the obvious next step is to create the instance and start it. There are many ways of doing that. One way, is using the AWS web console. Go to “Services” -> “EC2” and then “Launch Instance”. This will open a wizard dialog which allows you to choose the OS you want, network configuration, etc.

Another way is using docker-machine to create the instance which is described in the next section.

Creating and managing instances with docker-machine

Because I wanted to use Docker containers to run my demo web application, I thought it would also be useful to give Docker Machine a try. Docker Machine is a set of tools to create, delete, start, stop, and manage different kind of machines. It supports different backends like EC2, VirtualBox, etc.

What is great is that, once you get used to the commands, you will likely be able to use that common API across different providers. This tool comes with the “Docker Desktop” app (at least for OSX and Windows).

So, the first thing I did was to create ~/.aws/credentials to avoid having to pass ACCESS_KEY and SECRET_ACCESS_KEYas arguments to docker-machine commands.

[default] 
aws_access_key_id = XXX 
aws_secret_access_key = YYY

Now, let’s create the machine!

docker-machine create --driver amazonec2 --amazonec2-region us-east-2 --amazonec2-ami ami-0959e8feedaf156bf --amazonec2-instance-type a1.large --amazonec2-open-port 9000 --amazonec2-open-port 7777 --amazonec2-open-port 80 arm-a1-docker

A few details here:

By specifying –driver amazonec2 you are saying you will use EC2 instances. You can check here all possible arguments for this driver.
–amazonec2-ami ami-0959e8feedaf156bf is the AMI (Amazon Machine Image) to use. In this case, ami-0959e8feedaf156bf is Ubuntu 18.04 ARM 64 bit. You need to be sure that the selected AMI is supported in the specified –amazonec2-region as not all AMIs are possible for all regions as far as I understand.
–amazonec2-instance-type a1.large specifies the A1 large instance as already stated.
–amazonec2-open-port opens ports in the VM, in particular I am telling to open 80 and 7777 (for my web app) and 9000 for portainer.io tool. Note that when using docker-machine to create the instance, a security group called docker-machine will also be added in AWS with the ports 22 (ssh) and 2376 (docker) already open. You may want to protect the docker port.
Finally, arm-a1-docker is the name you give to the instance.

Once the create command is finished, you can check in the web console and confirm the instance is present:

That’s it, now you can use whatever docker-machine command you want, for example:

Running Docker in the created instance

When creating the instance with docker-machine and using the AMI I mentioned, docker comes already installed in the VM. The only thing I had to do was to connect via SSH into the server and then:

sudo usermod -aG docker ${USER}

After I did that, I was able to successfully run Docker hello world:

Running VA Smalltalk Web Application with Docker on the A1 Instance

The server and docker are now running. Deploying the real project is the only thing left to do. For that, I will use the same web app example “Seaside Traffic Light” that I have been using in my previous posts. There is even a working ARM docker example for it on Github:

Mariano Martinez Peck

@martinezpeck

My first #ARM 64 #Docker Swarm "2 nodes cluster" (@picocluster I will get to you sooner or later) with a #RaspberryPi 3 (Ubuntu 18.04) and a #Rock64 (Armbian). @traefik load balancing 10 #VASmalltalk 64 images running a #Seaside webapp. #Smalltalk Code: github.com/vasmalltalk/do…

15:15 PM - 08 May 2019

5 11

So all I have to do is to clone that repo, build the docker image, and start the Docker Swarm.

cd $HOME
mkdir Instantiations
cd Instantiations
git clone https://github.com/vasmalltalk/docker-examples.git
cd docker-examples/source/SeasideTrafficLights/Raspberry64/
docker build -f ./debian_slim_Dockerfile -t seaside-debian-slim.
./startSwarm.sh

And that’s it. We should now have a Docker Swarm running 10 VA Smalltalk containers (as specified in the docker-compose of that example) and you should be able to enter your web app at: http://YOUR_SERVER_IP/trafficlight. Pro tip: you can use the command docker-machine ip arm-a1-docker to get the external IP of the instance.

Managing Docker containers with Portainer

Portainer is a nice tool for managing and monitoring Docker containers from within a web console. The best thing in my opinion is how easy it is to run. Guess how? Of course, it’s another Docker container! All you need to do is:

docker volume create portainer_datadocker
docker run -d -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer

And then you can open a web browser on: http://YOUR_SERVER_IP:9000/

Conclusions

This was a very short introduction, but it shows how easy it is to deploy a VA Smalltalk project into AWS instances. Hopefully, there will be more experiments to come… Stay tuned!

"El Carrillon" project featured in the RaspberryPi magazine MagPi

Mariano Martinez Peck — Fri, 28 Feb 2020 15:05:38 +0000

Some time ago I blogged about a fantastic project Gerardo Richarte and I were doing with an 18-bell tower located in Argentina.

Back then, I showed some details of the architecture, code, and how I mocked-up “El Carrillon” with a homemade LED piano so I could test it at home.

Later on, I wrote another post, where you can see it running live and read some great news about the project (including an award!).

In this quick post, I am so happy to share with you some BIG NEWS:

El Carrillon is the first VASmalltalk IoT project to appear as a showcase in the official RaspberryPi magazine, The MagPi!. It was a pleasure for me to work on the project with Gerardo Richarte, and thanks to Instantiations for their support!

Thanks for re-sharing via twitter or linkedin post.

Liquid error: internal

El Carrillon: playing MIDI songs on an 18-bell tower with a Raspberry Pi and Smalltalk

Mariano Martinez Peck — Wed, 04 Dec 2019 11:49:56 +0000

Some time ago I blogged about a fantastic project Gerardo Richarte and I were doing with an 18-bell tower located in Argentina.

Back then, I showed some details of the architecture, code, and how I mocked-up “El Carrillon” with a homemade LED piano so I could test it at home.

In this post, you will see it running live and read some great news about the project!

Playing live at “La Fiesta Nacional de la Flor”

“El Carrillon” had to play not only during my LED piano testing, but also on the real hardware with real bells during a huge event last November called: “La Fiesta Nacional de la Flor”.

We first started testing at home with the real hardware, but without the bells to confirm the GPIOs were responding correctly:

Liquid error: internal

Once that was working, the next obvious step was to move to production and test with physical bells!! The mechanical/hardware part of some bells needed repair, so it took some time until we were able to test in-person. Once we did, everything worked as expected and without much trouble.

Deploying was really easy thanks to the “image” concept of Smalltalk and a few bash scripts that we had prepared.

The good news is that it worked and was a huge success, see below:

Liquid error: internal

Won 3rd place at the Innovation Technology Awards at ESUG 2019

We presented this project at the Innovation Technology Awards at ESUG 2019 held in Koln, Germany. Below is the teaser video we submitted before the competition:

Our efforts were worthwhile as we won 3rd place! Thanks to all that voted for us!

Liquid error: internal

Conclusion

I personally believe this was a great example of using IoT and Smalltalk. We used VAST (VA Smalltalk) for managing the GPIOs, running the web application, and all within a Raspberry Pi Zero.

VAST has great development and debugging tools, minimal Smalltalk images, good GPIO libraries, and excellent ARM support.

Let’s do more IoT projects with Smalltalk!!!

PS: The official Raspberry Pi twitter account did like our project! :)

Liquid error: internal

It’s 2019: stop reading text and start debugging!

Mariano Martinez Peck — Tue, 17 Sep 2019 11:31:41 +0000

If you are a programmer, I am sure you have found yourself many many times “debugging” a system by writing strings to the console / stdout / file. Those messages were usually something like “passed here!”, “false branch”, “now about to enter loop” and maybe even some “fuc..” or “WTF” too.

If you were “lucky” and your system went into production it’s very likely that you had to support and maintain it. How many times you found yourself “debugging” an error in production by reading hundreds of log files?

That’s not debugging. That’s text reading.

Imagine if you could live debug for real (with a real debugger) a deployed application? And that the application could even be running remotely. Or suppose that the application had an error and instead of writing a text-based stack trace, you could dump a snapshot of the execution stack (with all its variables and state) and then do a post-mortem analysis in your development environment (yes, again, with a real debugger)?

Is this how debugging will work in the future? No. I’ve been doing it already for years. Smalltalk has had this ability since at least 1995.

Don’t text the console. Inquire the process.

What is wrong with text reading and poor man’s debugging?

I could write a whole blog post about this, but here are some thoughts:

It’s a very unproductive and unpleasant task. It’s even worse if you have to do the stop – change (add logging) – start cycle.
It’s error prone. For example, you could have broken code while adding or removing your printing statements.
Whenever you finish your “debugging”, you must spend time rolling back all the changes. Even worse, you may forget some debugging code which makes it into production.
When you are looking at a log you are blind. You have no information about the execution stack (what was the state of this variable at this point? no clue..). At most, you could get “some” info thanks to the printing of the temp variables of each frame.
It’s not enough for mission critical systems. Many times, when there is an error and a system is down, it may be losing millions of dollars per hour.

Do you think that reading a log really helps? Maybe. But I hope I have already convinced you that we can do a lot better.

Demo: a simple web application

For this post I will be using a demo web application (written with Seaside framework) that represents a simple Traffic Light. When you click one of the three lights, the clicked one turns on and the others turn off:

One important detail is that, for demoing purposes, we put an error on one method that gets called when the green button is clicked:

toggleGreenLight
    | boolean allLights |
    boolean := true.
    allLights := Array new.
    allLights first turnOff.
    allLights second turnOff.
    self setGreenLight: boolean.

Did you spot it? No? Then keep reading ;) In this example we see one error in particular but everything I will show works exactly the same as if that would have been any kind a runtime error.

Now, we would need another machine that acts as the “production” server in which we could deploy our application and see how we can debug it. OK, for that, I will be using a Raspberry Pi that could be running anywhere in the world. This would also help me demonstrate how cool these unique debugging techniques are for IoT.

The debugging features I will be showing here are part of VASmalltalk but some of these features exist on other Smalltalk dialects.

Scenario 1: running on development

This is simple. He have the application running locally in our development environment and if we click the green button, we get a debugger:

In this situation there is not much magic. Well, at least for Smalltalkers. For non Smalltalkers there is some magic. For example you can inspect objects, send messages to them, evaluate code, change code and then just resume the execution. Yes, no stop-compile-start cycle. Live coding as they say. Do you now understand why we (Smalltalkers) love developing in the debugger?

Scenario 2: running on production with text logs

We are now running our application in our production machine (in this example a Raspberry Pi) and we will be using the classical log file for error handling.

Below you can see an extract of the generated log file (here it is called walkback.log). This is likely what you are most familiar with:

Taking a detailed look at the above log file illustrates the limitations I mentioned earlier in this post.

Scenario 3: running on production with binary stack dumping

Finally!!! The cool stuff begins :) In this case, when I clicked the green button the application did not generate a log file but a binary seaside.sdf. That file contains a snapshot of the execution stack at the moment the exception happened.

The magic is that later on I can then take any development environment, open a debugger, attach that seaside.sdf file and do a post-mortem analysis!

In this example, I copy the file from the server into my development machine and here is the result:

You can notice that on this debugger the Into, Over and all the navigation buttons are disabled. This is because this is a snapshot of a stack so you can’t really change it (wait for next section!). However, you can see how much friendlier this is than dealing with log files. Here you can see that I can inspect objects, I can see their state, etc…

At this point, the error should be clear: the variable allLights is empty and so sending first to it throws an exception. The fix here was to simply use an existing method turnAllOff:

toggleGreenLight

    self turnAllOff.
    self setGreenLight: true

Just to summarize: I have a local debugger opened with a snapshot of the execution stack when an exception was triggered on a remote server.

Scenario 4: running on production with live remote debugging

The previous binary stack dump is great for doing post-mortem analysis of production errors, but what if we want to live debug what is happening on the server? Or in a IoT board? Well, we can then use the remote debugger feature.

For this demo what I want to do is to halt the execution (like a breakpoint) at a certain place so that I could open a debugger and see what is going on. For that I added a Halt halt at the beginning of the method:

toggleGreenLight
    Halt halt. 
    self turnAllOff.
    self setGreenLight: true

The remote debugger works via TCP which means:

The “target” (remote system) must know the IP (or hostname) of the “development” environment.
We need certain port to be opened in the network.
We can remotely debug ANY remote system running anywhere in the globe. In this case, the Raspberry Pi is next to me, but it would the same if it is 14k km away.

The first thing I need to do is to tell my “development” environment to listen for possible remote connections:

Now, I can simply run the web application and click on the green button. The browser would then look like “waiting” on the callback of the button.

But…if I now go to my development machine…I see a debugger that came in!!!

Into, Over and the rest of the buttons are now available. Again, I can inspect objects, see their state and send messages to them. But in addition, I can change code and resume the execution (just as I did when developing on my local machine!). For example, let’s change the code so that now the green light is red:

And then I just click on Resume button. The debugger disappears and the web browser continues its rendering (but now, obviously, the green light is red):

Just to summarize: I have a local debugger opened with a live execution stack when an exception was triggered on a remote server. I even could modify code and resume the exception to continue its normal execution.

Too magical to believe it? Am I cheating?

The text and the screenshots might not be enough to properly show all these features, therefore, I recorded this short video to show things working live:

Other possible uses?

I have only shown a few scenarios, but there are more! Some others I am aware of:

Debugging Continues Integration test failures: sometimes a test failed on the CI but not on your local machine. So… we just added stack dumping to tests that fail. That way, you simply download the stack file and see what happened.
Sharing “errors” between the community: instead of sharing a plain string stack, you share stack dump.
Debugging AWS Lamba services.
Debugging on IoT.

Conclusions

We have been doing this for years. The binary stack dump and remote debugging of VASmalltalk dates from around 1995. Yes, 24 years ago. Pharo and GemStone Smalltalks both have similar features. In fact, I co-authored that Pharo development in 2012 which won first place at the ESUG Innovation Technology Awards.

So this is what “debug to the future ” looks like. The pun is doubly intended: Smalltalk has allowed this kind of debugging since ages ago, although it may look like a futuristic approach to programmers who haven’t tried it. Also, with the ideas shown in the post, you can see how you can have a stack trace now, but debug it much later, even in a different environment, kinda like suspended animation for computing processes.

Being a software professional also means caring about the tools you choose to use. I can use vim and know lots of great hotkeys, but I also have a nice monitor and a mouse, so why can’t I just enjoy and take some advantage of modernity?

I’d love for you to use Smalltalk (that would be very cool), but at the very least fight for having decent debugging capabilities in whatever language you are using.

People now use the trendy words “Technical debt”. Well, I promise that replacing a text editor and grep with a debugger does help to reduce it.

Finally, don’t stay with the idea that this is just something “cool”. It is cool, but it also brings a lot of value to businesses and customers.

It’s time to move forward.

PS: Thanks to all the BA-ST Slack group and Instantiations team that help me a lot with ideas, quotes, revisions and much more!

Getting Started with Nvidia Jetson Nano, TensorFlow and Smalltalk

Mariano Martinez Peck — Thu, 05 Sep 2019 11:40:50 +0000

On all my previous posts (like this one) you can see VASmalltalk running on any Raspberry Pi, on Rock64 and even on Nvidia Jetson TX2:

// Detect dark theme var iframe = document.getElementById('tweet-1141775466384306182-579'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1141775466384306182&theme=dark" }

In addition, you can also see previous posts where I show how to use TensorFlow from Smalltalk to recognize objects in images.

Last week, at ESUG 2019, I demoed a VA Smalltalk and TensorFlow project on an Nvidia Jetson Nano provided by Instantiations.

In this post, I will show you how to get started with the Jetson Nano, how to run VASmalltalk and finally how to use the TensorFlow wrapper to take advantage of the 128 GPU cores.

What do you need before starting

Below is the whole list of supplies I gathered:

NVIDIA Jetson Nano Developer Kit
MicroSD card: I got a Samsung 128GB U3
Power supply: you can choose either a limited USB or a much more powerful DC switching power supply. I got the latter.
Case (optional): I like metal cases so I got one with Power & Reset Control Switch.
Fan (optional): I could only find one fan that would fit on the Nano but it was very expensive.
Wireless Module (optional): the board does not come with built-in WiFi or Bluetooth so I decided to buy this module.

Assembling the Nano and related hardware

For this step, I first followed this short guide but finally moved to this one which was super detailed. I won’t repeat everything written there but instead I will add my own bits below.

I started by formatting the SD. For this, I always use “SD Card Formatter” program. Downloading the operating system image and flashing the SD was easy… But the first downside is that for the first boot you NEED an external monitor, keyboard and mouse. No way to do it headless :( After the first boot, you can indeed enable SSH and VNC, but not for the first time.

The next step was to assemble the Wifi and Bluetooth. It was not a walk in the park but not that difficult either. You need to disassemble the Nano a bit, connect some wires, etc:

// Detect dark theme var iframe = document.getElementById('tweet-1165271816224563200-706'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1165271816224563200&theme=dark" }

Something bad is that the board is configured to start by default with a USB power supply. In my case, I ordered a DC instead as it’s better if you want to take the maximum power. But…to tell the Nano whether to use USB or DC, you must change the jumper J48. But guess what? The development kit (100USD) does NOT even bring a single jumper. So you got your Nano and your DC power supply and you are dead, you can’t boot it. Seriously? (BTW, before you ask, no, I didn’t have a female-to-female cable with me that day as a workaround for the jumper)

The other complicated part was to assemble the case and the fan. For that, I needed to carefully watch this video a few times. Once it was built, it felt really solid and nice. BTW the case did come with a jumper for J48 which was really nice since that meant I could use the DC power supply.

The fan itself was also complicated. The Noctua NF-A4x20 5V PWM I bought wouldn’t fit easily. The NA-AV3 silicone anti-vibration mounts would not get through the holes of the Nano. And the screws for the fan provided by the case were too short. So I had to buy some other extra screws that were long enough.

When I was ready to try the fan, I powered it and nothing happened. I thought I did something wrong and I had to re-open the case a few times… painful process. I almost gave up, when I found some help over the internet. Believe it or not, you must run a console command in order to start the fan: sudo jetson_clocks. After that, it started working.

// Detect dark theme var iframe = document.getElementById('tweet-1168873407116644352-384'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1168873407116644352&theme=dark" }

Setting it to run headless

While in most boards and operating systems this is easy, on the Nano this is a challenging part. The SSH part is easy and you almost don’t need to do anything in particular. But for VNC… OMG…. I followed all the recommendations provided in this guide. In my case, I could never get the xrdp working… when it tries to connect from my Mac, it simply crashes…

As for VNC, after all the workarounds/corrections mentioned there, I was able to connect but the resolution was too bad (640×480). I spent quite some time googling until I found a workaround mentioned here. Basically, I did sudo vim /etc/X11/xorg.conf and I added these lines:

Section "Screen"
   Identifier    "Default Screen"
   Monitor        "Configured Monitor"
   Device        "Default Device"
   SubSection "Display"
       Depth    24
       Virtual 1280 800
   EndSubSection
EndSection

In other words, I needed to change the size of the Virtual Display that is used if no monitor is connected (by default it was 640×480).

After rebooting, I was finally able to get a decent resolution with VNC.

Installing VASmalltalk dependencies

This part was easy and I basically followed the bash script of a previous post:

# Install VA Dependencies for running headfull and VA Environments tool
sudo apt-get install --assume-yes --no-install-recommends \
  libc6 \
  locales \
  xterm \
  libxm4 \
  xfonts-base \
  xfonts-75dpi \
  xfonts-100dpi

# Only necessary if we are using OpenSSL from Smalltalk
sudo apt-get install --assume-yes --no-install-recommends \
  libssl-dev 

# Generate locales
sudo su
echo en_US.ISO-8859-1 ISO-8859-1 >> /etc/locale.gen
echo en_US.ISO-8859-15 ISO-8859-15 >> /etc/locale.gen
locale-gen
exit

Installing TensorFlow and VASmalltalk wrapper

The first thing you must do is to either build TensorFlow from scratch for Nvidia Jetson Nano with CUDA support or try to get a pre-build binary from somewhere. I am getting the latter using the following bash script:

mkdir tensorflow
cd tensorflow
wget https://dl.photoprism.org/tensorflow/nvidia-jetson/libtensorflow-nvidia-jetson-nano-1.14.0.tar.gz
tar xvzf libtensorflow-nvidia-jetson-nano-1.14.0.tar.gz
cd lib
ln -s libtensorflow_framework.so libtensorflow_framework.so.1

The symbolic link is a workaround because in the shared libraries that I downloaded, libtensorflow.so would depend on libtensorflow_framework.so.1 but the library that was shipped was libtensorflow_framework.so and so I made a symlink.

To install VASmalltalk and the TensorFlow wrapper, I followed the instructions from the Github repository. The only detail is that ARM64 VM will be shipped in the upcoming 9.2 ECAP 3….so send me a private message and I will send it to you until the release is public.

For the .ini file I added:

TENSORFLOW_LIB=/home/mpeck/Instantiations/tensorflow/lib/libtensorflow.so

The last bit is that TensorFlow needs help so that libtensorflow can find libtensorflow_framework. So what I did is to export LD_LIBRARY_PATH before starting the VASmalltalk image. Another possibility is moving the shared libraries to /usr/lib or /usr/local/lib. It’s up to you.

cd ~/Instantiations/VastEcap3_b437_7b7fc914f16f_linux/raspberryPi64/
export LD_LIBRARY_PATH=/home/mariano/Instantiations/libtensorflow-nvidia-jetson-nano-1.14.0.2/lib:$LD_LIBRARY_PATH
./abt64.sh

And all tests were green:

Confirming we are using GPU

By default, if a GPU is present (and the shared library was compiled with GPU support), TensorFlow will use GPU over CPU. From Smalltalk we can confirm this by checking the available TFDevice by inspecting the result of (TFSession on: TFGraph create) devices

You can then run a simple test like TensorFlowCAPITest >> testAddControlInput and see the log printed into the xterm. You should see that a GPU device is being used:

Using 128 GPU cores, TensorFlow and VASmalltalk to detect Kölsch beers with #esug19 pictures

OK. So we have TensorFlow running, all our tests passing and we are sure we are using GPU. The obvious next step is to run some real-world demo.

In a previous post you saw some examples of Object Detection. During the ESUG 2019 Conference I wanted to show this demo but instead of recognizing random objects on random images, I showed how to detect “beers” (Kölsch! we were at Cologne, Germany!) on the real pictures people uploaded to Twitter #esug19 hashtag.

// Detect dark theme var iframe = document.getElementById('tweet-1167438196038348801-607'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1167438196038348801&theme=dark" }

The code for that was fairly easy:

ObjectDetectionZoo new
    imageFiles: OrderedCollection new;
    addImageFile: '/home/mariano/Instantiations/tensorflow/esug2019/beer1.png';
    graphFile: '/home/mariano/Instantiations/tensorflow/frozen_inference_graph-faster_resnet50.pb';
    labelsFile: 'examples/objectDetectionZoo/mscoco_label_map.pbtxt';
    prepareImageInput;
    prepareSession;
    predict;
    openPictureWithBoundingBoxesAndLabel

And here are the results:

Conclusions

The Nvidia Jetson Nano does offer good hardware at a reasonable price. However, it’s much harder to setup than most of the boards out there. It’s the first board that takes me sooooo much time to get fully working.

But the worst part, in my opinion, is the state of Linux Tegra. Crashes everywhere and almost impossible to setup something as simple as VNC. I would really like to see a better/newer OS for the Nano.

Once all your painful setup is done, it works well and it provides nice GPU capabilities. We now have everything in place to start experimenting with it.

PS: Thanks Maxi Tabacman and Gera Richarte for doing a review of this post!

Object Detection with TensorFlow and Smalltalk

Mariano Martinez Peck — Mon, 19 Aug 2019 10:46:56 +0000

In a previous post we saw basic object recognition in images using Google’s TensorFlow library from Smalltalk. This post will walk you step by step through the process of using a pre-trained model to detect objects in an image.

It may also catch your attention that we are doing this from VASmalltalk rather than Python. Check out the previous post to see why I believe Smalltalk could be a great choice for doing Machine Learning.

Liquid error: internal

TensorFlow detection model Zoo

In this post, we will be again using a pre-trained model:

We provide a collection of detection models pre-trained on the COCO dataset, the Kitti dataset, the Open Images dataset, the AVA v2.1 dataset and the iNaturalist Species Detection Dataset. These models can be useful for out-of-the-box inference if you are interested in categories already in those datasets. They are also useful for initializing your models when training on novel datasets.

The original idea of using these models was written in this great post. Our work was also inspired by this and this Juypiter notebooks for the demo.

Designing the demo with a Smalltalk object-oriented approach

In the previous post you can see that all the demo was developed in the class LabelImage. While starting to implement this new demo we detected a lot of common behaviors when running pre-trained frozen prediction models. So… we first created a superclass called FrozenImagePredictor and changed LabelImage to be a subclass of it, overriding only a small part of the protocol.

After we finished the refactor it was quite easy to add a new subclass ObjectDetectionZoo. And all we needed to implement on that class was just 7 methods (and only 5 methods inLabelImage). So…as you can see, it’s quite easy now to add more and more frozen image predictors.

In the previous example (with LabelImage) we processed the “raw” results just as TensorFlow would answer it. However, with ObjectDetectionZoo the results were a bit more complex and in addition we needed to improve the readability of the information, for example, to render the “bounding boxes”. So we reified the TensorFlow results in ObjectDetectionImageResults.

These pre-trained models can answer the data for the “bounding boxes”. We were interested in seeing if we could render the image directly from Smalltalk and draw the boxes and labels. Again, time to reify that in ObjectDetectionImageRenderer.

To conclude, we have ObjectDetectionZoo which will run the model and answer ObjectDetectionImageResults and then delegate to ObjectDetectionImageRenderer to display and draw the results.

Running the examples!

To run the examples, you must first check the previous post to see how to install VASmalltalk and TensorFlow. After that, you can check the example yourself in the class comment of ObjectDetectionZoo.

This example runs the basic mobilenet_v1 net which is fast but not very accurate:

ObjectDetectionZoo new
    imageFiles: OrderedCollection new;
    addImageFile: 'examples\objectDetectionZoo\images\000000562059.jpg';
    graphFile: 'examples\objectDetectionZoo\ssd_mobilenet_v1_coco_2018_01_28\frozen_inference_graph.pb';
    prepareImageInput;
    prepareSession;
    predict;
    openPictureWithBoundingBoxesAndLabel

Then you can simply change the one line where you specify the graph to use rcnn_inception_resnet_v2 and you will see the results are much better:

In the tensorflow-vast repository we only provide a few frozen pre-trained graphs because they are really big. However, you can very easily download additional ones and use them. The only thing you must do is to uncompress the .tar.gz and simply change the one line where you specify the graph (graphFile:) to use rcnn_inception_resnet_v2 and you will see the results are much better:

You can see with mobilenet_v1 the spoon was detected as person, the apple on the left and the bowl were not detected and the cake was interpreted as a sandwich. With rcnn_inception_resnet_v2 all looks correct:

Something very cool from TensorFlow is that you can run multiple images in parallel on a single invocation. So here is another example:

ObjectDetectionZoo new
    imageFiles: OrderedCollection new;
    graphFile: 'z:\Instantiations\TensorFlow\faster_rcnn_inception_resnet_v2_atrous_coco_2018_01_28\frozen_inference_graph.pb';
    addImageFile: 'examples\objectDetectionZoo\images\000000463849.jpg';
    addImageFile: 'examples\objectDetectionZoo\images\000000102331.jpg';
    addImageFile: 'examples\objectDetectionZoo\images\000000079651.jpg';
    addImageFile: 'examples\objectDetectionZoo\images\000000045472.jpg';
    prepareImageInput;
    prepareSession;
    predict;
    openPictureWithBoundingBoxesAndLabel

Which brings these results:

As you can see here there are many different pre-trained models so you can use and experiment with any of those. In this post we just took 2 of them (mobilenet_v1 and rcnn_inception_resnet_v2) but you can try with anyone. All you need to do is to download the .tar.gz of that model, uncompress it, and specify the graph file with graphFile:.

Finally, you can also try with different pictures. You can try with any image of your own or try with the ones provided in the databases used to train these models (COCO, Kitti, etc.).

Conclusions and Future Work

We keep pushing to show TensorFlow examples from Smalltalk. In the future, we would really like to experiment with training models in Smalltalk itself. Particularly, we want to experiment with IoT boards with GPU (like Nvidia Jetson or similar).

It would also be interesting to try detecting objects on videos aside from pictures.

Finally, thanks to Gera Richarte for the help on this work and to Maxi Tabacman for reviewing the post.

Recognizing objects in images with TensorFlow and Smalltalk

Mariano Martinez Peck — Wed, 07 Aug 2019 11:50:02 +0000

In this post, I will be showing a simple example of object recognition in images using the TensorFlow library from Smalltalk.

Doing Machine Learning without Python? With Smalltalk? Seriously?

Whenever you start entering the world of AI and Machine Learning you will notice immediately that Python has been widely accepted as the “default” programming language for these topics.

I am not against Python and I believe that people are using it for a reason. However, I do believe that providing alternatives is a good thing, too. And Smalltalk could be that alternative you are looking for.

Last week, my smalltalker friend Maxi Tabacman told me:

Smalltalk is for complex models and Machine Learning is complex.

Apart from Maxi’s point, there is another reason to have it in Smalltalk. If you are working in a Smalltalk project, you are likely managing complex data. This means you would probably benefit from applying Machine Learning. So, why would you work outside of your Smalltalk environment and model to apply it?

Then, even Gilad Bracha told me:

A civilized front end for TensorFlow would be great.

As for what that means – well that’s a long discussion :)

A long time ago, Gera “Richie” Richarte said:

Smalltalk is an excellent environment for learning and experimentation. It’ll be a great place to start playing with Machine Learning.

And he did eventually start playing with ML and Smalltalk a few years ago. He started developing a TensorFlow wrapper for Cuis Smalltalk. Later on, this binding was ported to Pharo Smalltalk too and now finally to VASmalltalk. So…as you can see.. as Serge Stinckwich said:

All Smalltalk unite for Machine Learning and Deep Learning!

This project is not just a binding/wrapper. It tries (as it is still evolving and improving) to provide a much richer API and take advantage of the unique features of Smalltalk.

Liquid error: internal

Preparing the example

In this post I will be using the VASmalltalk TensorFlow wrapper but a similar approach could be done with the other Smalltalk dialects. The first thing you must do is to download TensorFlow shared libraries, install the Smalltalk wrapper and configure it. All the details are explained in the official docs. To confirm everything is working with the wrapper, you can run the 250+ unit tests we have (not all of them are currently passing in this early preview of the VASmalltalk version).

Liquid error: internal

You will see that TensorFlow examples are loaded within the standard load process. This means you should have the LabelImage class loaded. You may want to check its class comment for explanations and details.

LabelImage example shows how you can use a pre-trained TensorFlow network to recognize objects in images. As this is a pre-trained model, you will need its proto buffers file (.pb). This example also needs a “labels” file. But don’t worry, we have already included everything necessary in examples/labelImage. So… all you need to do is copy the directory examples from the git root directory to your Smalltalk image working directory.

Running the examples!

Single image – Inception V3

The first example runs with this image:

And runs an Inception V3 model. You can read more info about it here.

|results|
results := LabelImage new
    imageSize: 299@299;
    imageFiles: OrderedCollection new;
    graphFile: 'examples\labelImage\inception_v3_2016_08_28_frozen.pb';
    labelsFile: 'examples\labelImage\labels.txt';
    addImageFile: 'examples\labelImage\grace_hopper.jpg';
    prepareImageInput;
    prepareSession;
    predict.
Transcript cr.
results do: [ :eachImage |
    (eachImage first: 3) do: [:each |
        Transcript 
            show: each key;
            show: ' %';
            show: ((each value * 100) roundTo: 0.01) asString;
            cr.
    ]
].

And that should print the following on the Transcript:

military uniform %92.5
mortarboard %1.61
academic gown, academic robe, judge's robe %0.44

Single image – Mobilenet V1

The Mobilnet model doesn’t give results as good as the Inception V3 but it runs much faster.

You can run this code:

((LabelImage new
        imageSize: 224@224;
        graphFile: 'examples\labelImage\mobilenet_v1_1.0_224_quant_frozen.pb';
        labelsFile: 'examples\labelImage\labels.txt';
        addImageFile: 'examples\labelImage\grace_hopper.jpg';
        prepareImageInput;
        prepareSession;
        predict)
            first
                first: 10)
                    collect: [:each | each key -> ((each value * 100) roundTo: 0.01) asString]

And it will print:

 ('military uniform' -> '39.73'
 'Windsor tie' -> '24.14'
 'bow tie, bow-tie, bowtie' -> '10.52'
 'bulletproof vest' -> '4.59'
 'mortarboard' -> '3.88' 
 'bolo tie, bolo, bola tie, bola' -> '1.69'
 'suit, suit of clothes' -> '1.21' 
 'bearskin, busby, shako' -> '1.21'
 'microphone, mike' -> '0.87' 
 'wig' -> '0.74')

Important: note that in both models I am specifying a different image size: 299 for Inception V3 and 224 for Mobilenet V1. This is because the image size is dependent on the pre-trained model we are using. You must read the documentation in order to know which value to use.

Multi image – Inception V3

In the previous examples we saw only one image processing at a time. However, TensorFlow allows us to run multiple images in parallel. This is what the example below does:

|results|
results := LabelImage new
    imageSize: 299@299;
    imageFiles: OrderedCollection new;
    graphFile: 'examples\labelImage\inception_v3_2016_08_28_frozen.pb';
    labelsFile: 'examples\labelImage\labels.txt';
    addImageFile: 'examples\labelImage\grace_hopper.jpg';
    addImageFile: 'examples\labelImage\ski.jpg';
    addImageFile: 'examples\labelImage\Carrillon-Escobar-1.jpg';
    addImageFile: 'examples\labelImage\byteSmalltalk.jpg';
    prepareImageInput;
    prepareSession;
    predict.
Transcript cr.
results do: [ :eachImage |
    (eachImage first: 3) do: [:each |
        Transcript 
            show: each key;
            show: ' %';
            show: ((each value * 100) roundTo: 0.01) asString;
            cr.
    ].
    Transcript cr. 
].

Here are the results. Can you match the picture to the Transcript results? :)

Important: note that in this case there is still only ONE Smalltalk process doing a single (well, a few) FFI call to TensorFlow… the exact same FFI calls as it would do for a single-image scenario. The magic is in TF, as it will parallelize the processing of those images. Depending on the hardware, how TF was built, your configuration, etc., it may be using only CPUs or also GPUs.

How to train your dragon

This post was about a pre-trained net that we simply downloaded. But obviously, the more interesting and complex scenario is to develop and train your own model. That will be a topic for a future post. But in the meantime, you can get a taste of it by checking the snippets below.

The first one is nice and simple:

initializeForwardGraph
   | input |
   input := graph placeholder: 'Input' type: TF_Tensor typeFloat.
   activation := (weights * input) sigmoid.

The following one is a bit more complicated but you can see how you can nest things to build the net:

initializeInferenceGraph
        hidden1 := graph
                fromBlock: [:image |
                        input := image.
                        (image * weights1 + biases1) rectified]
                named: 'layer1'.
        hidden2 := graph fromBlock: [
                (hidden1 * weights2 + biases2) rectified] named: 'layer2'.
        prediction := graph
                fromBlock: [
                        netInput := hidden2 * weights3 + biases3.
                        netInput softmax findMaxOn: 1 asInt32Tensor]
                named: 'layer3'.

Ready to train your own?

Now imagine… TensorFlow + Smalltalk + ARM + GPU

As you can see in previous posts like this one and this one, I have been building TensorFlow on different ARM-based SoB (Raspberry Pi, Rock64, etc). I succeeded on both ARM and ARM64. And VASmalltalk supports both too.

Liquid error: internal

So…why did I do this? One obvious reason is that soon I want to test an ARM-based board with good GPU…like Nvidia Jetson Nano (128 GPU cores) or XT2 (256 cores). That is….I want TensorFlow running and taking advantage of all those cores!

Liquid error: internal

Conclusions

Only time will tell if doing Machine Learning and Deep Learning with Smalltalk is a good idea. From my side, even if I am far from being an expert in this area, I will do my best to make it happen.

Thanks to Gera Richarte and Koan-Sin TanThanks for pushing these examples and to Maxi Tabacman for reviewing the post.

The battle continues: Build TensorFlow C Binding for Raspberry Pi in 2019

Mariano Martinez Peck — Mon, 29 Jul 2019 13:39:23 +0000

In a recent post I showed up how challenging it still is to build TensorFlow C bindings for Raspberry Pi and other SBCs (Single Board Computer) and the lack of pre-build binaries.

As you could read, I was successful with certain approach (cross-compiling with a RaspberryPi-only script) but I wasn’t yet able to compile on the target (RaspberryPi 3 in this example) and I still had quite some questions open.

In this post I will show some more successful attempts as well as answer some of my previous questions.

Successful attempt: build from scratch on Raspberry Pi 3B+ and Raspbian Buster

I suspected one of the reasons of why I failed to do this on the first place was the amount of the swap space as well as free hard disk space.

It was the perfect excuse to buy another 64GB fast micro SD, try Raspbian Buster and start from a fresh install. So, that’s what I did.

If you remember from the previous post, the first thing you needed to do was to install Bazel. In this case, the first thing I did is to give it a lot of swap space. Instead of 2GB as I did before, I now assigned 8GB.

Next, remember you don’t need any version of Bazel. The version that TensorFlow would need. One open question I had from previous post was “how do I know that?” And here is the answer: you must check the file tensorflow/tensorflow/tools/ci_build/install/install_bazel.sh

For example, for 1.13.1 you can see:

# Select bazel version.
BAZEL_VERSION="0.20.0"

set +e
local_bazel_ver=$(bazel version 2>&1 | grep -i label | awk '{print $3}')

if [[ "$local_bazel_ver" == "$BAZEL_VERSION" ]]; then
  exit 0
fi

OK, so I started compiling Bazel following the instructions of the previous post and I found out problems with the Java VM. I am sorry, but I didn’t write down which was the exact issue. But it was an error clearly related with some Java compilation that I did NOT have with my previous attempt.

My first step was to check JVM versions (java --version) of this Raspbian Buster vs the Raspbian Stretch I used before. On the latter it showedjava version “1.8.0_65″ while in the former openjdk 11.0.3 2019-04-16. OK…so Stretch came with Java 8 while Buster with 11.

This is when I imagined that maybe Bazel could be build only with a given Java version in particular. Which one? No clue (tell me if you do). So what I did on my fresh Raspbian Buster is to install java 8 too:

sudo apt-get install openjdk-8-jdk

After that, you can check which are the alternatives:

$ update-java-alternatives -l
java-1.11.0-openjdk-armhf      1111       /usr/lib/jvm/java-1.11.0-openjdk-armhf
java-1.8.0-openjdk-armhf       1081       /usr/lib/jvm/java-1.8.0-openjdk-armhf

Obviously, I didn’t want to change the default Java version for my whole OS, so I just made the following workaround:

env BAZEL_JAVAC_OPTS="-J-Xms384m -J-Xmx1024m" \
JAVA_TOOL_OPTS="-Xmx1024m" \
JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk-armhf" \
EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" \
bash ./compile.sh

Basically, I am just telling Bazel to use Java 8. After that, it could finish correctly. The next step was to build TensorFlow.

I started my build process but I hit another compilation issue related to AWS which I could workaround by just telling to NOT compile AWS….

Finally, after 13 hours (yes, that’s the expected time) I could successfully build it:

// Detect dark theme var iframe = document.getElementById('tweet-1152631270377218048-680'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1152631270377218048&theme=dark" }

The final Bazel compilation script for TensorFlow was like this:

env JAVA_HOME="/usr/lib/jvm/java-1.8.0-openjdk-armhf" \
bazel --host_jvm_args=-Xmx1024m --host_jvm_args=-Xms384m build \
--config=noaws \
--config opt --verbose_failures --local_resources 1024,1.0,1.0 \
--copt=-mfpu=neon-vfpv4 \
--copt=-ftree-vectorize \
--copt=-funsafe-math-optimizations \
--copt=-ftree-loop-vectorize \
--copt=-fomit-frame-pointer \
--copt=-DRASPBERRY_PI \
--host_copt=-mfpu=neon-vfpv4 \
--host_copt=-ftree-vectorize \
--host_copt=-funsafe-math-optimizations \
--host_copt=-ftree-loop-vectorize \
--host_copt=-fomit-frame-pointer \
--host_copt=-DRASPBERRY_PI \
//tensorflow/tools/lib_package:libtensorflow

Interesting points:

I added the JAVA_HOME workaround for Java 8.
I removed the original --jobs=3 that I used before because didn’t seem to change much.
I added the --config=noaws workaround for AWS.
As I knew it would take a lot of time to compile I wanted to double check the CPU temperature, so every in a while I run /opt/vc/bin/vcgencmd measure_temp and check its results.

Once it finishes, you may want to copy the resulting files somewhere. In my case I did:

sudo cp bazel-bin/tensorflow/libtensorflow.so /usr/local/lib/
sudo cp bazel-bin/tensorflow/libtensorflow_framework.so /usr/local/lib/

Failed attempt (but with learnings!): build from scratch on Raspberry Pi 3B+ and Ubuntu Server 18.04 and aarch64 (ARM 64 bits)

Once I had ARM 32 bits working, my next step was to try on aarch64 (ARM 64 bits). At the time of this writing, there is no official 64 bits Raspbian version. It’s not the first time I want to run something on the Pi3B+ with aarch64 so I already had a micro SD with Ubuntu Server 18.04 up and running.

I started eating my own dog food and here it comes the first issue: the way to change the swap partition is not the same with Raspbian and Ubuntu. So I followed this guide for Ubuntu and assigned also 8GB.

Second, it seems you need python installed. No, this Ubuntu did not have any python installed. I then understood why some blog posts started with a “first, install dependencies..” and provided below line:

sudo apt-get install gcc g++ swig build-essential openjdk-8-jdk python zip unzip

I let you decide if you want Python 2.x or 3x. As we found out before, the JDK version is also important.

Anyway, after those 2 issues, I was able to compile and run bazel. However, as soon as I tried to run the previous bazel script for TensorFlow I got a compilation error saying that --copt=-mfpu=neon-vfpv4 was not a recognized option:

Thanks to freedomtan he told me that I don’t need all those extra --copts and --host_copts (they are for complicated Raspian environ). So the bazel script should then be something simpler:

bazel --host_jvm_args=-Xmx1024m --host_jvm_args=-Xms384m build \
--config=noaws \
--config opt --verbose_failures --local_resources 1024,1.0,1.0 \
//tensorflow/tools/lib_package:libtensorflow

I thought this was going to work, but after many hours it just didn’t finish. It somehow hung. I didn’t fight any longer because this Ubuntu installation was never really stable for me. So I will try this again anytime soon with a fresh install of Debian Buster or something for aarch64.

Successful attempt: convinced someone else to build it!

In the previous post I commented about this Github repo providing binaries for TensorFlow and Raspberry Pi. The drawback was that all binaries they ship were Python only and not the C shared library.

After a nice discussion with the author, he now build it for ARM and ARM64!! And seem to have included it as part of his release process. But that’s not all. It seems that even if Google itself provides official Python wheels for Raspberry Pi, there are many people that still uses his builds. Why? He explains himself:

The difference between my Wheel file and the official Wheel is the following two points.

* Official Wheel is disabled by default on Tensorflow Lite. However, my wheel is enabled by default.

* Tuned to 2.5 times the performance of the official Tensorflow Lite.

It is 1. that is considered a problem by engineers around the world.

You can read more in above link and see how to get those binaries! BTW, the same author also provides binaries of Bazel for ARM!

Future attempts (yes, the journey is not over!)

Cross-compile myself (any combination for this purpose) using these instructions. The main issue with this kind of compilation is that for some reason the resulting shared library seems linked with a concrete glibc version. Therefore, its usage is limited because it will only work with operating systems matching that glibc (you can try multi-glibc in the same host but it seems a pain). If you know how we can fix this, please let us know.
Cross-compile myself using this other instructions which seems interesting too. Unfortunately the author does not show too much interest in providing C bindings.
Try again aarch64 compilation on the Pi but with a different OS (not Ubuntu).

Challenge Accepted: Build TensorFlow C Binding for Raspberry Pi in 2019

Mariano Martinez Peck — Wed, 17 Jul 2019 14:26:27 +0000

Believe me. Setting up the environment and building TensorFlow C binding for Raspberry Pi is more complicated than training a neural network that makes me rich by robo-trading assets.

Motivation

As SBCs (Single Board Computer) get more and more powerful and cheap, the more likely we will want to run some more heavy computation on them. People like to use terms like “Edge Computing”, “Embedded HPC or ML” or similar terms.

Something quite common between all these different SBCs alternatives is the use of ARM processors plus some type of GPU.

A classical example of this heavy computation is AI (Artificial Intelligence) and ML (Machine Learning). In this area, one of the most used and accepted library is Google’s TensorFlow. Such library is written in Python. However, there are also pre-build official binaries for C, Java and Go.

The C API is commonly used for binding to other languages via FFI (Foreign Function Interface). From my point of view, that’s a critical binary.

Currently, I am developing/testing a VASmalltalk binding that uses the C library via FFI. I tested on Linux x64, I tested on Windows and then I wanted to try in Raspberry Pi 3B+, Pine64, Nvidia Jetson Nano, etc… Why? Because I truly believe that this “embedded ML” (or whatever you call it) has value. Running machine learning algorithms in a 35USD machine seems interesting to me.

// Detect dark theme var iframe = document.getElementById('tweet-1149286273661710336-29'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1149286273661710336&theme=dark" }

So…what happened? I simply went to TensorFlow’s official website and look for the shared library. Guess what? There was none. Zero. Null. Nil. No binaries for any kind of ARM board. I was so surprised that I asked in StackOverflow.

I understand that there are plenty of boards out there each with different hardwares, softwares, drivers, operating systems, etc. But I was expecting at least to have it for some very common ones like Raspberry Pi and Nvidia Jetson Nano.

Anyway…this is how my journey started. I am not sure if my writings would be useful for others, but at least for my future me, I am sure they will.

The next sections are sorted in the order I look for the solutions.

DISCLAIMER: I am NOT a TensorFlow expert. So if you have any feedback, please share!

Failed attempt 1: install Python version and extract the shared library from there

With some recent TensorFlow version, Raspberry Pi / Raspbian is officially supported (I think >= 1.9). However, the only “binaries” available are Python wheels. I suspected Python would be using C underneath so I install the Python version directly on my Pi following the official instructions using pip:

pip3 install --user --upgrade tensorflow # install in $HOME

I then look into the intalled directory and found some shared libraries!

cd /usr/local/lib/python3.5/dist-packages/tensorflow/python
ls -lah _pywrap_tensorflow_internal.so
-rwxr-xr-x 1 root staff 154M Jul  1 09:32 _pywrap_tensorflow_internal.so

But guess what? _pywrap_tensorflow_internal.so is not the same as the shared library we need for the C binding (libtensorflow.so.1.14.0)

I kept looking and then I found an installation with Docker. But again, only possible to build binaries for Python, not for C.

After all my failed attempts, I opened a case on Github as a “feature request”.

Failed attempt 2: looking for non official pre-build binaries

The obvious next step was… “OK, if Google doesn’t do it, then someone else must”. I mean…. Smalltalk is not the only one wanting to bind against the C library, right?

Long short story, I found none. I found this one, but it was only python builds (but he said he may try providing shared libraries…so stay tunned!). I then found this one that would work at least for Nvidia Jetson Nano (but I don’t have the Nano with me yet). I found another one, but again only Python.

So….in conclusion, I didn’t find the shared library for the Raspberry Pi anywhere. If you are aware of something, please let me know. What was worst was that most answers were “you better compile it yourself”. That didn’t sound too bad…I mean…sure, why not? Until I checked the official size of the Linux x64 shared library and the libtensorflow.so was 216MB. WHATTTTTTTTTT? At that moment I thought “OK, this is not gonna be easy”.

Abandoned attempt: build from scratch on the Pi

My next obvious step was to try to build from scratch on the Pi. For that, I based my work on this very helpful step by step guide. However, time has passed since that guide was written, TensorFlow become “a bit easier” to build on the Pi and so some instructions from it are not necessary anymore. In addition, I found my own problems that were not addressed there.

I recommend you read that guide first and then continue here. Below is what I ended up doing, which is similar to that guide.

Before getting started, some important tips I recommend:

Have many free GBs in your Pi disk.
Be sure to NOT be running anything heavy on the Pi (shutdown X, VNC, docker, whatever that can use CPU or memory).
Run the build from ssh.
Use tmux or similar tool because the process takes hours (many hours) and so you will likely want to power off your development machine and check Pi status the next morning.
Use heat sinks in your Pi if you don’t want to burn it.

Building the builder: bazel

The first thing is that to build TensorFlow you need the bazel tool. Of course: sudo apt-get install bazel, right? hahahahahahah. LOL. I wish it was that simple. Once again, it looks there is no bazel package ready to install on the Pi. So you must first compile it. OK…this thing is becoming meta. I need to build the builder…what’s next? to compile the Linux kernel in which i will build the builder? …

Now…to compile either bazel or TensorFlow, in both cases, the 1GB RAM of your Pi won’t be enough. So you must increase the swap space. In the mentioned guide it mounts an external USB stick / hard disk / etc. In my case, I just increased the swap partition from the SD card to 2GB. But people recommend more….like 8GB (but I didn’t have that much free):

sudo vim /etc/dphys-swapfile # change CONF_SWAPFILE to 2000
sudo /etc/init.d/dphys-swapfile stop
sudo /etc/init.d/dphys-swapfile start
free -m # confirm we have now 2000mb

IMPORTANT: Whether you success or not with bazel and TensorFlow compilation, its VERY important that you put back the original swap space size (CONF_SWAPFILE) when you are done. Else, you will ruin the SD lifespan.

The compilation of bazel can take a few hours. Once I finished and started to compile TensorFlow, I got a wonderful message:

Please downgrade your bazel installation to version 0.21.0 or lower to build TensorFlow!

Are you kidding me?????? I spent hours compiling the wrong bazel version? FUC… Is there a way to know in advance which bazel version each TensorFlow version needs? I have no clue. If you know, please tell me. Anyway, I started over with the version it needed for the version of TensorFlow I wanted (1.13.1):

mkdir bazel
cd bazel 
wget https://github.com/bazelbuild/bazel/releases/download/0.21.0/bazel-0.21.0-dist.zip
unzip bazel-0.21.0-dist.zip
env BAZEL_JAVAC_OPTS="-J-Xms384m -J-Xmx1024m" \
JAVA_TOOL_OPTS="-Xmx1024m" \
EXTRA_BAZEL_ARGS="--host_javabase=@local_jdk//:jdk" \
bash ./compile.sh
sudo cp output/bazel /usr/local/bin/bazel
cd ..
rm -rf bazel*

The Java options for the memory (the 1024 is because the Pi 3B+ has 1GB RAM) are necessary because else compilation just fails (thanks freedomtan for the help). And no, it doesn’t fail with a nice “Out of Memory” but some kind of random error. I reported that into a Github issue.

The other necessary part --host_javabase=@local_jdk//:jdk. I don’t even remember why…it simply wouldn’t work without that.

If you succeed on doing this, save that bazel binary everywhere! don’t loose it hahahaha. Again, if you know somewhere where I can find bazel pre-build binaries for the Pi, please let me know.

Building TensorFlow

The first steps are trivial:

git clone --recurse-submodules https://github.com/tensorflow/tensorflow.git
cd tensorflow
git checkout v1.13.1
./configure

The ./configure will ask you a few questions about what support you want to add to the TensorFlow compilation you are about to do. The answers will depend on the hardware you are targeting. For Raspberry Pi I think it’s OK to simple answer false to all of them:

Watching at the questions, you may get an idea what you will eventually answer for Nvidia Jetson, Parallella Board, etc. And yes, I would like to see if it works on the Parallella Board:

// Detect dark theme var iframe = document.getElementById('tweet-1147132349450203136-374'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1147132349450203136&theme=dark" }

Finally, time to run compilation. No, don’t grab a beer, you will end drunk. No, don’t take coffee…you will drink so much caffeine that you will not be able to sleep for a whole week.

bazel --host_jvm_args=-Xmx1024m --host_jvm_args=-Xms384m build \
--config opt --verbose_failures --jobs=3 --local_resources 1024,1.0,1.0 \
--copt=-mfpu=neon-vfpv4 \
--copt=-ftree-vectorize \
--copt=-funsafe-math-optimizations \
--copt=-ftree-loop-vectorize \
--copt=-fomit-frame-pointer \
--copt=-DRASPBERRY_PI \
--host_copt=-mfpu=neon-vfpv4 \
--host_copt=-ftree-vectorize \
--host_copt=-funsafe-math-optimizations \
--host_copt=-ftree-loop-vectorize \
--host_copt=-fomit-frame-pointer \
--host_copt=-DRASPBERRY_PI \
//tensorflow/tools/lib_package:libtensorflow

Some interesting points about that:

Most of the --copt and --host_copt were not identified by me. Again, thanks freedomtan.
I already explain why the Java memory arguments.
--verbose_failures is useful if our build fails to get some description of what went wrong.
--local_resources helps specify “how much hardware resources” to use.
For me, it was still failing to build because of low resources. So I ended up adding --jobs=3 which minimizes the use of resources (but will take longer, obviously). I got this from a StackOverflow.
It’s interesting to note that building a Python Wheel or a shared library is almost the same process. The only change is that instead of //tensorflow/tools/lib_package:libtensorflow (for .so) you use //tensorflow/tools/pip_package:build_pip_package to get a Wheel. That’s why I was kindly asking those already providing Wheels, to also provide shared libraries.
This process will take many many hours (in my case it took more than 20). So, go to sleep and check the next morning.

This should work. However, as it was taking too much time, I continued looking for other alternatives and I never really let the process to finish. So I can’t confirm it works. And now my SD doesn’t have free space and I already got a working .so (next section). If you try it and it works, let me know! Otherwise, I guess I will try again in the near future.

Final attempt: cross-compiling

While I was waiting the compilation on the Pi to finish and suffering by watching its green led constantly turned on for hours and hours, I continued looking for more alternatives. By chance, I arrived to an official link that showed how to cross-compile TensorFlow for the Pi. (I should have seen this before! hahahahaha)

Just to understand how difficult it is to have all the environment setup ready, imagine that the cross-compile procedure is to use Docker and start off from an existing image they provide…

The procedure looked very simple: install docker and then run one shell line:

tensorflow/tools/ci_build/ci_build.sh PI \
    tensorflow/tools/ci_build/pi/build_raspberry_pi.sh

Cool. That sounded magical. Too good to be true. I was then ready to take advantage of all my 8 CPU cores and 16 GB RAM of my MBP. Unfortunately, the process never finished well for me. Each run would fail at a different place and the explanation was never clear. Again, I opened a case on Github but no response so far.

I was about to abandon all my attempts for TensorFlow on ARM / SBC. But I had one last idea: try again this cross-compilation with Docker but now on a Linux virtual machine that I had with Linux Mint 18.3. Of course, this VM was never gonna be as fast as doing it directly in my host (OSX), but it should still be much faster than doing it on the Pi.

Call it a miracle or not, but after a few hours, that DID WORK. I successfully got the .so, moved it into the Pi and then run my tests. Everything was working:

// Detect dark theme var iframe = document.getElementById('tweet-1151100675104940032-44'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1151100675104940032&theme=dark" }

Conclusion

I hope Google would officially ship C binaries at least for the most common SBC like Raspberry Pi or Jetson Nano. If not that, I hope some of those people already compiling Wheels for Raspberry could compile shared libraries too.

Finally, nevertheless, I think it was worth for me learning the low level details of a build from scratch. Why? Because there are many boards I would like to experiment with: Rpi3 but with a ARM 64 OS (Armbian, Ubuntu Server 18.04, etc), Rpi4, Pine64, Jetson Nano, etc. We can even test on an Nvidia Jetson TX2!!! And for all these cases, I won’t be able to use the cross-compile alternative out-of-the-box because that was intended for the Pi only.

// Detect dark theme var iframe = document.getElementById('tweet-1141775466384306182-133'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1141775466384306182&theme=dark" }

I hope I could have helped someone else aside from my future me. If you have any kind of feedback please share!