DEV Community: Jérémie Laval

Using DirectManipulation with WPF

Jérémie Laval — Sun, 31 Mar 2019 02:16:00 +0000

Or said differently, how to enjoy smooth pinch-to-zoom gestures on a legacy platform that’s not UWP (no pun intended).

If you have done any UI developement on Windows, chances are you have used a ScrollViewer at some point. If you use the UWP version of ScrollViewer you will notice that in addition to, well, scroll events the control also natively handles zooming. In the case of WPF however, this capability is completely absent.

The main reason for this is that WPF was created at a time where neither precision trackpad nor touchscreen were ubiquitous (touch was just starting with Windows 8 after all). Rather it was designed with stylus support in mind and some initial iteration for inking recognition. Since WPF has been put on ice until recently, it never got a fresh coat of paint on its input APIs and things have indeed quite changed since then.

For starter, the low-level input strategy on Windows is now different. Under the hood of Win32, input events coming from trackpad, touch, stylus and others are now all unified under a single WM_POINTER event type where previously they were separate. If the app doesn’t have support for it however, the OS will try to convert those events and gestures into a somewhat equivalent combination that the app might recognize.

For our case (zooming and more precisely pinch to zoom gestures), that compatibility combination is a “Ctrl + Mouse Wheel Up/Down” event which is a standard pattern that apps could implement before. The trouble is that the mouse wheel event that’s sent is only in increment/decrement of a default value which means you can only implement “jagged” zoom behaviors (least you feel like animating everything yourself). This would look a little bit like this:

container.PreviewMouseWheel += HandleZoomGesture;

void HandleZoomGesture (object sender, MouseWheelEventArgs e)
{
    // Input emulation will make it look like the Ctrl key is pressed
    // when converting the pinch-to-zoom gesture
    if ((Keyboard.Modifiers & ModifierKeys.Control) == 0)
        return;

    var zoomDelta = ZoomGestureDelta * (((double)e.Delta) / Mouse.MouseWheelDeltaForOneLine);
    SetZoom (currentZoom + zoomDelta);
    e.Handled = true;
}

This compatibility behavior might work for your case and, as an example, is what some browsers still implement (at least Firefox as I’m writing this) or also the Visual Studio text editor. If you would like something better however read on.

As it turns out, WPF did have its input subsystem updated to use this new unified WM_POINTER event type but it was never enabled by default and still today sits under an appcontext switch (Switch.System.Windows.Input.Stylus.EnablePointerSupport). Despite using the new events, that updated stack also willingly filters out scenario that are not strictly coming from touch or stylus thus completely bypassing the added support for precision trackpad.

Fortunately, there is a way out. Likely to allow external apps that are not UWP based to still benefit from this gesture support (aka browsers), a generic low-level API does exists going by the name of DirectManipulation. This API operates at the HWND level and works by having you pass on those WM_POINTER messages who then get processed on a different thread with DirectManipulation handling all the heavy lifting of recognizing gestures and calculating the right transformation to apply.

It’s normally supposed to be used together with DirectComposition which is another low-level API to interact with Windows compositor but you can still make it work manually by simply extracting the values you need (aka content scale).

The DirectManipulation API is not provided out of the box on the framework but you can use SharpDX which has bindings for it or a tool like tlbimp on the IDL of the library. Or you can trust me with this binding I made.

For your convenience, here is a “minimal” wrapper class, inspired from Chrome’s usage, that implements and use a subset of DirectManipulation so that you can plug it into your WPF application (including the support to understand WM_POINTER events):

using System;
using System.Runtime.InteropServices;
using System.Windows;
using System.Windows.Interop;
using DirectManipulation;

class PointerBasedManipulationHandler : IDirectManipulationViewportEventHandler, IDisposable
{
    const int ContentMatrixSize = 6;
    // Actual values don't matter
    static tagRECT DefaultViewport = new tagRECT { top = 0, left = 0, right = 1000, bottom = 1000 };

    HwndSource hwndSource;
    IDirectManipulationManager manager;
    IDirectManipulationViewport viewport;
    Size viewportSize;

    uint viewportEventHandlerRegistration;
    float lastScale;
    float[] matrix = new float[ContentMatrixSize];
    IntPtr matrixContent;

    float lastTranslationX, lastTranslationY;

    public PointerBasedManipulationHandler ()
    {
        matrixContent = Marshal.AllocCoTaskMem (sizeof (float) * ContentMatrixSize);
    }

    public event Action<float> ScaleUpdated;
    public event Action<float, float> TranslationUpdated;

    public HwndSource HwndSource {
        get => hwndSource;
        set {
            var first = hwndSource == null && value != null;
            var oldHwndSource = hwndSource;
            if (oldHwndSource != null)
                oldHwndSource.RemoveHook (WndProcHook);
            if (value != null)
                value.AddHook (WndProcHook);
            this.hwndSource = value;
            if (first && value != null)
                InitializeDirectManipulation ();
        }
    }

    IntPtr Window => hwndSource.Handle;

    void InitializeDirectManipulation ()
    {
        this.manager = (IDirectManipulationManager)Activator.CreateInstance (typeof (DirectManipulationManagerClass));
        var riid = typeof (IDirectManipulationUpdateManager).GUID;
        riid = typeof (IDirectManipulationViewport).GUID;
        this.viewport = manager.CreateViewport (null, Window, ref riid) as IDirectManipulationViewport;

        var configuration = DIRECTMANIPULATION_CONFIGURATION.DIRECTMANIPULATION_CONFIGURATION_INTERACTION
            | DIRECTMANIPULATION_CONFIGURATION.DIRECTMANIPULATION_CONFIGURATION_SCALING
            | DIRECTMANIPULATION_CONFIGURATION.DIRECTMANIPULATION_CONFIGURATION_TRANSLATION_X
            | DIRECTMANIPULATION_CONFIGURATION.DIRECTMANIPULATION_CONFIGURATION_TRANSLATION_Y
            | DIRECTMANIPULATION_CONFIGURATION.DIRECTMANIPULATION_CONFIGURATION_TRANSLATION_INERTIA;
        viewport.ActivateConfiguration (configuration);
        viewportEventHandlerRegistration = viewport.AddEventHandler (Window, this);
        viewport.SetViewportRect (ref DefaultViewport);
        viewport.Enable ();
    }

    public void Dispose ()
    {
        viewport.RemoveEventHandler (viewportEventHandlerRegistration);
        Marshal.FreeCoTaskMem (matrixContent);
        HwndSource = null;
    }

    public void SetSize (Size size)
    {
        this.viewportSize = size;
        viewport.Stop ();
        var rect = new tagRECT {
            left = 0,
            top = 0,
            right = (int)size.Width,
            bottom = (int)size.Height
        };
        viewport.SetViewportRect (ref rect);
    }

    // Our custom hook to process WM_POINTER event
    IntPtr WndProcHook (IntPtr hwnd, int msg, IntPtr wParam, IntPtr lParam, ref bool handled)
    {
        if (msg == WM_POINTERDOWN || msg == DM_POINTERHITTEST) {
            var pointerID = GetPointerId (wParam);
            var pointerInfo = default (POINTER_INFO);
            if (!GetPointerInfo (pointerID, ref pointerInfo))
                return IntPtr.Zero;
            if (pointerInfo.pointerType != POINTER_INPUT_TYPE.PT_TOUCHPAD &&
                pointerInfo.pointerType != POINTER_INPUT_TYPE.PT_TOUCH)
                return IntPtr.Zero;

            viewport.SetContact (pointerID);
        } else if (msg == WM_SIZE && manager != null) {
            if (wParam == SIZE_MAXHIDE
                || wParam == SIZE_MINIMIZED)
                manager.Deactivate (Window);
            else
                manager.Activate (Window);
        }
        return IntPtr.Zero;
    }

    void ResetViewport (IDirectManipulationViewport viewport)
    {
        viewport.ZoomToRect (0, 0, (float)viewportSize.Width, (float)viewportSize.Height, 0);
        lastScale = 1.0f;
        lastTranslationX = lastTranslationY = 0;
    }

    public void OnViewportStatusChanged ([In, MarshalAs (UnmanagedType.Interface)] IDirectManipulationViewport viewport, [In] DIRECTMANIPULATION_STATUS current, [In] DIRECTMANIPULATION_STATUS previous)
    {
        if (previous == current)
            return;

        if (current == DIRECTMANIPULATION_STATUS.DIRECTMANIPULATION_READY)
            ResetViewport (viewport);
    }

    public void OnViewportUpdated ([In, MarshalAs (UnmanagedType.Interface)] IDirectManipulationViewport viewport)
    {
    }

    public void OnContentUpdated ([In, MarshalAs (UnmanagedType.Interface)] IDirectManipulationViewport viewport, [In, MarshalAs (UnmanagedType.Interface)] IDirectManipulationContent content)
    {
        content.GetContentTransform (matrixContent, ContentMatrixSize);
        Marshal.Copy (matrixContent, matrix, 0, ContentMatrixSize);

        float scale = matrix[0];
        float newX = matrix[4];
        float newY = matrix[5];

        if (scale == 0.0f)
            return;

        var deltaX = (newX - lastTranslationX);
        var deltaY = (newY - lastTranslationY);

        bool ShallowFloatEquals (float f1, float f2)
            => Math.Abs (f2 - f1) < float.Epsilon;

        if ((ShallowFloatEquals (scale, 1.0f) || ShallowFloatEquals (scale, lastScale))
            && (Math.Abs (deltaX) > 1.0f || Math.Abs (deltaY) > 1.0f)) {
            TranslationUpdated?.Invoke (-deltaX, -deltaY);
        } else if (!ShallowFloatEquals (lastScale, scale)) {
            ScaleUpdated?.Invoke (scale);
        }

        lastScale = scale;
        lastTranslationX = newX;
        lastTranslationY = newY;
    }

    #region Native methods
    const int WM_POINTERDOWN = 0x0246;
    const int DM_POINTERHITTEST = 0x0250;
    const int WM_SIZE = 0x0005;

    static readonly IntPtr SIZE_MAXHIDE = new IntPtr (4);
    static readonly IntPtr SIZE_MINIMIZED = new IntPtr (1);

    uint GetPointerId (IntPtr wParam) => (uint)(unchecked((int)wParam.ToInt64 ()) & 0xFFFF);

    [DllImport ("user32.dll", EntryPoint = "GetPointerInfo", SetLastError = true)]
    internal static extern bool GetPointerInfo ([In] uint pointerId, [In, Out] ref POINTER_INFO pointerInfo);

    [StructLayout (LayoutKind.Sequential, CharSet = CharSet.Unicode)]
    internal struct POINTER_INFO
    {
        internal POINTER_INPUT_TYPE pointerType;
        internal uint pointerId;
        internal uint frameId;
        internal POINTER_FLAGS pointerFlags;
        internal IntPtr sourceDevice;
        internal IntPtr hwndTarget;
        internal POINT ptPixelLocation;
        internal POINT ptHimetricLocation;
        internal POINT ptPixelLocationRaw;
        internal POINT ptHimetricLocationRaw;
        internal uint dwTime;
        internal uint historyCount;
        internal int inputData;
        internal uint dwKeyStates;
        internal ulong PerformanceCount;
        internal POINTER_BUTTON_CHANGE_TYPE ButtonChangeType;
    }

    [StructLayout (LayoutKind.Sequential, CharSet = CharSet.Unicode)]
    internal struct POINT
    {
        internal int X;
        internal int Y;
    }

    internal enum POINTER_INPUT_TYPE : uint
    {
        PT_POINTER = 0x00000001,
        PT_TOUCH = 0x00000002,
        PT_PEN = 0x00000003,
        PT_MOUSE = 0x00000004,
        PT_TOUCHPAD = 0x00000005
    }

    [Flags]
    internal enum POINTER_FLAGS : uint
    {
        POINTER_FLAG_NONE = 0x00000000,
        POINTER_FLAG_NEW = 0x00000001,
        POINTER_FLAG_INRANGE = 0x00000002,
        POINTER_FLAG_INCONTACT = 0x00000004,
        POINTER_FLAG_FIRSTBUTTON = 0x00000010,
        POINTER_FLAG_SECONDBUTTON = 0x00000020,
        POINTER_FLAG_THIRDBUTTON = 0x00000040,
        POINTER_FLAG_FOURTHBUTTON = 0x00000080,
        POINTER_FLAG_FIFTHBUTTON = 0x00000100,
        POINTER_FLAG_PRIMARY = 0x00002000,
        POINTER_FLAG_CONFIDENCE = 0x000004000,
        POINTER_FLAG_CANCELED = 0x000008000,
        POINTER_FLAG_DOWN = 0x00010000,
        POINTER_FLAG_UPDATE = 0x00020000,
        POINTER_FLAG_UP = 0x00040000,
        POINTER_FLAG_WHEEL = 0x00080000,
        POINTER_FLAG_HWHEEL = 0x00100000,
        POINTER_FLAG_CAPTURECHANGED = 0x00200000,
        POINTER_FLAG_HASTRANSFORM = 0x00400000,
    }

    internal enum POINTER_BUTTON_CHANGE_TYPE : uint
    {
        POINTER_CHANGE_NONE,
        POINTER_CHANGE_FIRSTBUTTON_DOWN,
        POINTER_CHANGE_FIRSTBUTTON_UP,
        POINTER_CHANGE_SECONDBUTTON_DOWN,
        POINTER_CHANGE_SECONDBUTTON_UP,
        POINTER_CHANGE_THIRDBUTTON_DOWN,
        POINTER_CHANGE_THIRDBUTTON_UP,
        POINTER_CHANGE_FOURTHBUTTON_DOWN,
        POINTER_CHANGE_FOURTHBUTTON_UP,
        POINTER_CHANGE_FIFTHBUTTON_DOWN,
        POINTER_CHANGE_FIFTHBUTTON_UP
    }
    #endregion
}

To use the helper, instantiate it somewhere and pass it the HwndSource where your zoomable content is hosted. You can do so using the PresentationSource.AddSourceChangedHandler method like so:

PointerBasedManipulationHandler manipulationHandler = new PointerBasedManipulationHandler ();

PresentationSource.AddSourceChangedHandler (container, HandleSourceUpdated);

void HandleSourceUpdated (object sender, SourceChangedEventArgs e)
{
    if (manipulationHandler != null && e.NewSource is System.Windows.Interop.HwndSource newHwnd)
        manipulationHandler.HwndSource = newHwnd;
}

Finally, subscribe to the handler’s ScaleUpdated / TranslationUpdated events to react to DirectManipulation changes. For instance you can use a ScaleTransform as your container RenderTransform and set the corresponding values when you receive the event. Also don’t forget to set the size of your canvas by calling the SetSize method (for instance wired up from a SizeChanged event).

Now (at the end of the post) here is the final important caveat. If you want any of this DirectManipulation integration to work you are going to have to do one of two things. Because of the way DirectManipulation employs input delegation, you cannot have the legacy WPF stylus input stack run inside your application (which is the default). As such you have two options:

Disable native stylus/touch support and instead only rely on DirectManipulation: set the Switch.System.Windows.Input.Stylus.DisableStylusAndTouchSupport=true switch in you app.config or programmatically.
Enable the pointer-based stylus/touch input stack which is compatible with DirectManipulation: set the Switch.System.Windows.Input.Stylus.EnablePointerSupport=true switch in your app.config or programmatically (early on).

XML parsing in the Roslyn era

Jérémie Laval — Wed, 21 Mar 2018 13:00:00 +0000

Roslyn was a revolution in the .NET world when it came out. For the first time we had Microsoft release their compiler implementation of C# under an open-source license. Not only that, the way in which this compiler had been constructed would open the doors to numerous improvements in the ecosystem in the forms of:

A complete API to manipulate code as data
A compiler platform that is easy to hack on
A backend that is adapted for editor/language service usage

This enabled faster evolutions of the C# language, a plethora of tools and analyzers that can fully understand your code and intellisense that’s better than ever.

In my day to day job I work on visual tools at Microsoft for our Xamarin platform. As part of this we integrate with both Visual Studio Mac and Visual Studio where we, of course, use Roslyn for most of our user code interactions.

However, those visual tools all have another thing in common: they work on some dialect of XML (be it XAML, Android layout XML, XML resources or others). In a lot of ways working with that XML medium in the context of an editor presents the same challenges than Roslyn solved for code:

You need high-fidelity parsing of your document preserving user modifications
You need to react to and create changes applied to specific text subsets
You need a representation that is easy to manipulate and pass around

Existing XML solutions (System.Xml, System.Xml.Linq, …) tend to fall short because they weren’t created with an editor usage in mind. Their use case is more about exposing a document content than caring much about the way it’s written and evolved.

So basically what we need is Roslyn but for XML. Fortunately Kirill thought the same and a few years ago started working on bringing an existing morsel of Roslyn, namely the VB XML literal parser, and transform it into a fully fledged XML parser and syntax representation.

That project is (originally) named XmlParser

In a nutshell, XmlParser brings the Roslyn SyntaxTree AST model (immutable, full-fidelity, error-tolerant) and the parsing infrastructure that can create it to the point that its usage should feel familiar to someone who is used to work with Roslyn already.

As part of ongoing work, I spent a bit of time myself during the last few weeks bringing some improvements to the project to make it even better for an editor use case:

Proper red-green node separation which is a key to Roslyn performance while retaining immutability
Parsing incrementality: the ability to reuse existing syntax tree nodes when only a portion of the text buffer has changed
The start of proper syntax factories and modification APIs
More Roslyn code import where it made sense (common utilities, caching, …)

Albeit part of this is still a work-in-progress, we published updated NuGet packages (now under the GuiLabs.Language.Xml NuGet ID) with the latest code so that you can try it for yourself.

Computation Expressions in C# using async/await

Jérémie Laval — Thu, 11 Jan 2018 13:30:21 +0000

I have had some fun already using C# 7.1 new customizable async/await pipeline to do interesting things:

In this post I’ll be describing how I took on those past experiences and generalized the core idea of the second entry to provide a somewhat equivalent construct to F# computation expressions in C# by (ab)using the async/await machinery.

If you have never heard of F# computations expressions before I recommend reading the F# for fun and profit serie on it. Otherwise the TL;DR is that, without the monadic baggage, computation expressions essentially allow you to customize how a series of “statement” are chained together and plug your own logic in-between.

A famous example of such computation expression is the async workflow that lead to the creation of C# async/await in the first place (C# async/await being a more specialized/optimized form of its computation expression parent).

In a way what I’m describing in this post is basically how we can manage to return to the original definition of F# computation expressions using the more specialized C# async/await interface that was derived from it.

If you want to jump straight to the code grab it on GitHub. The repository also contains a Workbook file with a few examples utilizing the API so that you can get a feel of what it allows you to do.

For the others, here is the example given in the repository README of what the API looks like:

Option<int> TryDivide (int up, int down)
    => down == 0 ? None<int>.Value : Some.Of (up / down);

class MaybeBuilder : IMonadExpressionBuilder
{
    IMonad<T> IMonadExpressionBuilder.Bind<U, T> (IMonad<U> m, Func<U, IMonad<T>> f)
    {
        switch ((Option<U>)m) {
          case Some<U> some: return f (some.Item);
          case None<U> none:
          default: return None<T>.Value;
        }
    }

    public IMonad<T> Return<T> (T v) => Some.Of (v);
    public IMonad<T> Zero<T> () => None<T>.Value;
    // We don't have optional interface methods in C# quite yet
    public IMonad<T> Combine<T> (IMonad<T> m, IMonad<T> n) => throw new NotSupportedException ();
}

// Returns `Some 15`
ComputationExpression.Run<int, Option<int>> (new OptionExpressionBuilder (), async () => {
    var val1 = await TryDivide (120, 2);
    var val2 = await TryDivide (val1, 2);
    var val3 = await TryDivide (val2, 2);
    return val3;
})

// Returns `None`
ComputationExpression.Run<int, Option<int>> (new OptionExpressionBuilder (), async () => {
    var val1 = await TryDivide (120, 2);
    var val2 = await TryDivide (val1, 0);
    var val3 = await TryDivide (val2, 2);
    return val3;
})

In this snippet I’m lifting the example of the Maybe/Option monad I had talked about previously and encoding it via the computation expression mechanism to arrive at the same result. In that case the await keyword is used to represent the equivalent F# let! keyword which is a shortand for the Bind method of the computation expression builder (Bind is the core monadic operation that allows all those “statements” to be chained together).

To implement a custom computation expression you have to do two things:

Define a type that implements the empty IMonad interface. This simply acts as a marker to be able to use the custom async machinery of the library.
More interestingly, provide an implementation of the IMonadExpressionBuilder interface which is where you can put the code to customize the execution behavior of the computation expression block.

In the example above, the main customization is in the Bind method where we grab the result of executing the previous statement and then depending on the outcome either pass the result down the chain or short-circuit everything by returning None.

Another case where C# did a specialized implementation of what in F# is also encoded as a computation expression is IEnumerable<T> state machines via the yield operator.

With the library we can instead express this using the Combine method of the builder to achieve something like this (lifted from the examples workbook):

using static Neteril.ComputationExpression.ComputationExpression;
// The Enumerable monad is given in the library
using Neteril.ComputationExpression.Instances;

var result = ComputationExpression.Run<int, EnumerableMonad<int>> (new EnumerableExpressionBuilder (), async () => {
    var item = await (EnumerableMonad<int>)new [] { 1, 2, 3 };
    var item2 = await (EnumerableMonad<int>)new [] { 100, 200 };
    // We want back a enumeration containing the concatenation of (item, item2, item1 * item2)
    // for all successive values of item1 and item2
    await Yield (item);
    await Yield (item2);
    return item * item2;
});
string.Join (", ", result.Select (i => i.ToString ()));
// => [1, 100, 100, 1, 200, 200, 2, 100, 200, 2, 200, 400, 3, 100, 300, 3, 200, 600]

In this new sample, the monad is implemented in a way that the first await (again equivalent to F# let!) binds to each successive element of the given collections. The result is a EnumerableMonad<T> that is itself a IEnumerable<T> (i.e. you can further use LINQ on it).

Because in this case there is no way to hijack the C# yield operator directly (like F# would) this is instead represented here with the expression await Yield for intermediary values and return for the last one.

Here is the implementation of the builder for the sample to work (note the now implemented Combine method):

public class EnumerableExpressionBuilder : IMonadExpressionBuilder
{
    IMonad<T> IMonadExpressionBuilder.Bind<U, T> (IMonad<U> m, Func<U, IMonad<T>> f)
    {
       var previousEnumerableMonad = (EnumerableMonad<U>)m;
       return new EnumerableMonad<T> (previousEnumerableMonad.SelectMany (u => (EnumerableMonad<T>)f (u)));
    }

    IMonad<T> IMonadExpressionBuilder.Return<T> (T v) => new EnumerableMonad<T> (Enumerable.Repeat (v, 1));
    IMonad<T> IMonadExpressionBuilder.Zero<T> () => new EnumerableMonad<T> (Enumerable.Empty<T> ());

    IMonad<T> IMonadExpressionBuilder.Combine<T> (IMonad<T> m, IMonad<T> n)
    {
       var enumerableMonad1 = (EnumerableMonad<T>)m;
       var enumerableMonad2 = (EnumerableMonad<T>)n;
       // Simply use LINQ Concat method
       return new EnumerableMonad<T> (enumerableMonad1.Concat (enumerableMonad2));
    }
}

Interestingly of the early writings I remember reading about Monads and how it related to C#, the LINQ SelectMany operator was given as an example of a Bind operation in the monadic sense. With this we have the proof it’s indeed true.

The actual code enabling all of this is a fairly straightforward async method builder implementation. If you have read any of my previous articles I mentioned at the beginning of this post, it’s essentially doing the same thing with a bunch more hacks to cope with the added indirection of generic types in the IMonad<T> interface.

It also peeks a bit deeper into the internal structure of the async state machine generated by the compiler via the Machinist class so that it can control the actual stepping and awaiter values that are stored in it. This for instance allows the same state machine to be “rewinded” which is essential to support cases like the above Enumerable sample.

ActivityTask, an helper for async/await on Android

Jérémie Laval — Mon, 22 May 2017 13:30:21 +0000

While at I/O last week, we happened to attend the Architecture Component talk on Android lifecycle (which I recommend you watch). While the solutions presented there are definitely interesting and, in some cases, map to patterns we already have in .NET, it definitely resonated with us present on how those Android lifecycle nitpicks make one specific C# feature more cumbersome to use: async/await.

With async/await, the two main nitpicks that are pertaining to Android developers are:

Because of Android resource system works, a change of configuration (e.g. screen rotation) will recreate Activity instances by default
Because await is oblivious to an Activity lifecycle, it may execute continuations while in an undesired state causing an IllegateStateException

Note that we already introduced something that partially address those qualms in the form of the ActivityController with the added bonus of providing convenient asynchronous wrapper methods for StartActivityForResult-based workflows.

In this case, based on some other fun I had previously, I figured there was potentially a way to leverage C# 7 new customizable async state machines drivers to achieve something that could help mitigate those two issues without requiring too much changes in your code.

Enter ActivityTask (also on NuGet).

This little library has two main primitives:

ActivityScope allows to track the most recent instance of one of your Activity subclass so that potential underlying re-creations are transparent to you
ActivityTask acts as a standard Task return value in an async method but customize the state machine driver (in cooperation with ActivityScope) to make continuation scheduling aware of the activity lifecycle.

Under the hood, ActivityScope simply registers a listener at the Application level to listen to global activity lifecycle events. When it detects the activity it’s tracking is about to die, it will mark it so that when it’s respawned it can re-associate with it. Because it implements an implicit conversion operator to Activity, you can pass the scope wherever an activity instance is needed to ensure you always use a valid value.

As for ActivityTask the implementation is pretty boring (it pretty much defers to a TaskCompletionSource), the interesting portion is with ActivityScopeMethodBuilder that drives the async state machine. For all intent and purpose, it will behave like the default driver for Task. However, thanks to the extra ActivityScope method argument, it will additionally make sure that any continuation is only executed when the activity tracked by the scope is in a usable state at that moment. If not, it will simply keep the continuation queued up until the tracked activity is resumed.

To see how all of this fits together, here is the code for the activity of the test app in the GitHub repository:

[Activity(Label = "ActivityTaskTest", MainLauncher = true, Icon = "@mipmap/icon")]
public class MainActivity : Activity
{
    static bool launched = false;

    protected override async void OnCreate(Bundle savedInstanceState)
    {
        base.OnCreate(savedInstanceState);

        // Set our view from the "main" layout resource
        SetContentView(Resource.Layout.Main);

        if (!launched)
        {
            launched = true;
            using (var scope = ActivityScope.Of(this))
                await DoAsyncStuff(scope);
        }
    }

    TextView MyLabel(Activity activity) => activity.FindViewById<TextView>(Resource.Id.myLabel);

    async ActivityTask DoAsyncStuff(ActivityScope scope)
    {
        await Task.Delay(3000); // Medium network call
        MyLabel(scope).Text = "Step 1";
        await Task.Delay(5000); // Big network call
        MyLabel(scope).Text = "Step 2";
    }
}

This example simulates launching a series of asynchronous operations the very first time an activity is created. Beforehand though, it creates an ActivityScope to track the lifetime of the current activity and pass it down. Between each asynchronous substep, the activity instance is used to fetch the label onscreen and update its text.

The idea is to trigger a damaging event during one of those Task.Delay calls. Some that you can test are rotating your device (thus killing and re-creating the activity) or pressing the home button to pause the activity and reopen it after the delay expired.

For instance, if rotating the screen after “Step 1” is shown, you should see the original text of the label briefly re-appears (because the layout was inflated from scratch) and soon after see “Step 2” be set meaning the async method used the correct new activity instance to locate the label.

If pausing the activity after “Step 1” is shown, resuming the activity after the second delay expires should result in “Step 2” being displayed immediately as the callback got executed during the resume process instead of running while the activity was in the background.

Diverting functions in Windows with IAT patching

Jérémie Laval — Fri, 23 Dec 2016 10:00:21 +0000

Recently I was stuck with a blocking issue that was putting a feature I worked on in jeopardy. To summarize the problem, the entire feature relied on a specific way a library is initialized, the library in question providing two different functions (that we will call Init and InitExtended) to initialize itself.

The code responsible for this initialization hardcodes calling one of the variant and is in a DLL that is not modifiable. To make things harder, the two initialization functions I mentioned have slightly different parameters which complicated things as it can’t be a simple function pointer swap.

For exploitation purposes, an important aspect is that the initialization function is not actually referenced directly in the DLL but instead it is located dynamically using the standard LoadLibrary/GetProcAddress combo.

When something is using LoadLibrary, the obvious thing to try first is to take advantage of the DLL loader lookup mechanism and provide a modified DLL that can be loaded instead of the original one. In this case though, LoadLibrary was used with a hardcoded path in a system directory which made this option impossible.

Which leaves trying to use GetProcAddress to masquerade things up. To do this there is a more exotic technique called IAT patching that relies on the way the PE (Portable Executable) format work for dynamic libraries.

What’s in a DLL

Like most other dynamically loadable libraries format, a DLL contains more than just the code that’s executed. The file is split between different sections that references several other type of data in addition to the code section.

For instance the data section will contain a bunch of interesting tables, things like strings literals, the constant data in the program, thread local storage and so on.

When a DLL is loaded, those tables and sections are mapped into the calling process memory space and are accessible from user code although in practice you will always be using some system API instead of querying those parts of the memory yourself.

The Import Table

The table we are going to be interested in here is the Import Address Table (abbreviated IAT). The role of this table is to list all the external functions a DLL references from other libraries in a dictionary form keyed by library name (e.g. “user32.dll”) and values as function pointer thunks.

For instance, here is a snippet of the content of this table from a random DLL (the hint/ordinal values are used by the loader for faster processing):

$ pedump -I freetype.dll 

=== IMPORTS ===

MODULE_NAME HINT ORD FUNCTION_NAME
KERNEL32.dll 1cb GetCurrentThreadId
KERNEL32.dll 15b FlsSetValue
KERNEL32.dll 18c GetCommandLineA
KERNEL32.dll 425 RtlUnwindEx
KERNEL32.dll 2d3 HeapAlloc
KERNEL32.dll 208 GetLastError
KERNEL32.dll 2da HeapReAlloc
KERNEL32.dll 2d7 HeapFree
KERNEL32.dll ee EncodePointer
KERNEL32.dll 15a FlsGetValue
KERNEL32.dll 159 FlsFree
KERNEL32.dll 480 SetLastError
KERNEL32.dll 158 FlsAlloc
KERNEL32.dll 4c0 Sleep

In our case, both of LoadLibrary and GetProcAddress would be in this table keyed under the library they come from aka kernel32.dll.

The reason for this table is explained by the way dynamic libraries and the Windows DLL loader works. When compiling code to a DLL that depends on some other libraries, the actual memory layout where the dependencies are going to be loaded is not known in advance. Maybe that dependency has already been loaded by the program, maybe the DLL will be the one to load it, in any case the real locations for all those functions will ultimately be determined by the running context.

Thus, when some code references an external function, what the compiler will do is insert a call instruction that is not pointing directly to the real address of the function but instead will load the address pointed to by the thunk I mentioned earlier from the Import Address Table.

By doing so, the code contained inside the DLL is always valid and the Windows loader simply has to modify one piece of the overall loaded DLL (the IAT thunk) to point to the right function address at runtime instead of going throughout the entire code section and patch one-by-one every call instructions.

Below is an example of how the same DLL we saw earlier is calling the GetCommandLineA Win32 function contained in kernel32. Notice how the address for the call instruction is retrieved (qword) from a location in the data segment (ds) and how the disassembler is smart enough to name the thunk with a imp_ prefix:

And here is a view of the actual array containing all the thunks for kernel32 functions that will be patched by the Windows loader when the DLL is processed at run-time:

Patching the Import Address Table

Reminder: we are trying to influence a program execution by manipulating GetProcAddress so that instead of calling an Init function it calls a different InitExtended function.

Now that we know how external functions are called it’s easy to see that taking control of this IAT indirection is the key to do what we want. By patching the IAT thunk for the function we want to modify, we ensure the relevant code will be redirected to use our modifications.

Provided you have already the ability to load custom code in the process via another mean (if not lookup DLL injection techniques), the recipe to hijack a IAT thunk is as follow:

Create your custom function in your own code that mimics the prototype of the real one
Prevent or delay the codepath you are trying to change from accessing the function before your patching is done
Retrieve a pointer to the module (containing all the tables) of the DLL/executable that you are trying to hijack
Locate the Import Address Table
Iterate over it to find the right thunk
Overwrite the value given by the DLL loader with your own custom function address

Below is the translated and annotated version of this recipe in C using the imagehlp library (that defines all the right C structs to modelize the IAT table layout):

// Our custom Init function that will now internally
// call the InitExtended function we wanted to use
static void MyCustomInitFunction (/* parameters */)
{
  InitExtended (/* other parameters */);
}

// Our hooked version of GetProcAddress with the
// same prototype (and same calling convention)
static FARPROC MyGetProcAddress (HMODULE hModule, LPCSTR lpProcName)
{
  if (strcmp (lpProcName, "Init") == 0)
    return (FARPROC)MyCustomInitFunction;
  return GetProcAddress (hModule, lpProcName);
}

static BOOL HackGetProcAddress ()
{
  // Get the handle to the module we want to divert 
  HMODULE module = GetModuleHandle (L"TheDllToHijack.dll");
  if (module == NULL)
    return FALSE;

  // Get a reference to the import table to locate the kernel32 entry
  ULONG size;
  PIMAGE_IMPORT_DESCRIPTOR importDescriptor = (PIMAGE_IMPORT_DESCRIPTOR)ImageDirectoryEntryToDataEx (module, TRUE, IMAGE_DIRECTORY_ENTRY_IMPORT, &size, NULL);

  // In the import table find the entry that corresponds to kernel32
  BOOL found = FALSE;
  while (importDescriptor->Characteristics && importDescriptor->Name) {
    PSTR importName = (PSTR)((PBYTE)module + importDescriptor->Name);
    if (_stricmp (importName, "kernel32.dll") == 0) {
      found = TRUE;
      break;
    }
    importDescriptor++;
  }
  if (!found)
    return FALSE;

  // We use this value as a comparison
  PROC baseGetProcAddress = (PROC)GetProcAddress (GetModuleHandle (L"kernel32.dll"), "GetProcAddress");

  // From the kernel32 import descriptor, go over its IAT thunks to
  // find the one used by the rest of the code to call GetProcAddress
  PIMAGE_THUNK_DATA thunk = (PIMAGE_THUNK_DATA)((PBYTE)module + importDescriptor->FirstThunk);
  while (thunk->u1.Function) {
    PROC* funcStorage = (PROC*)&thunk->u1.Function;
    // Found it, now let's patch it
    if (*funcStorage == baseGetProcAddress) {
      // Get the memory page where the info is stored
      MEMORY_BASIC_INFORMATION mbi;
      VirtualQuery (funcStorage, &mbi, sizeof (MEMORY_BASIC_INFORMATION));

      // Try to change the page to be writable if it's not already
      if (!VirtualProtect (mbi.BaseAddress, mbi.RegionSize, PAGE_READWRITE, &mbi.Protect))
        return FALSE;

      // Store our hook
      *funcStorage = (PROC)MyGetProcAddress;

      // Restore the old flag on the page
      DWORD dwOldProtect;
      VirtualProtect (mbi.BaseAddress, mbi.RegionSize, mbi.Protect, &dwOldProtect);

      // Profit
      return TRUE;
    }

    thunk++;
  }

  return FALSE;
}

Conclusion

This was an example of how you can use IAT patching to circumvent legacy code without having to recompile or otherwise damage the original binary.

This technique is well known so if you want to do something more complicated/reliable you can always use an existing library like EasyHook or something as full-featured as Microsoft’s Detours.

References:

Custom controls in the Xamarin Android Designer

Jérémie Laval — Sat, 17 Dec 2016 16:00:21 +0000

aka an introductory look at the internals of our Android ecosystem

In the latest stable release of the Xamarin platform, one specific feature I worked on is custom controls support in the Android designer (both Mac with Xamarin Studio and Windows on Visual Studio).

Our iOS designer has long enjoyed this kind of support, actually from day 1. Being part of that team at the time, I can confirm we were quite excited (and relieved!) to see that pie chart renders on stage when Miguel introduced the feature at Evolve 2013:

However, because the rendering strategy is quite different between the two designers, attaining the same level of support for custom controls on Android was much more challenging.

In this post I want to highlight how the Android designer and, more generally, Xamarin.Android work and how implementing custom controls fitted into this.

Designer Architecture

The Android designer is roughly split into two main parts. The first one is written in C# and constitute the bulk of the code. It’s a cross-IDE library that implements all the visual pieces of a modern designer like a surface canvas or a property panel and all the plumbing to make those work including a full DOM for Android layouts.

The second part is implemented in Java and, unlike the previous part that is run together with Xamarin Studio or Visual Studio, the Java layer is run into its own dedicated process. To summarize what it does, let’s say that it mostly handles the heavy-lifting of parsing resources and rendering Android layouts.

That last duty specifically (rendering layouts) is actually accomplished by using an existing Google library called layoutlib. It’s the same underlying library that powers the designer in Android Studio. Its goal is to take an XML layout, inflate it, layout it and render it to an image buffer.

Layoutlib is pretty nifty, it works by reusing an existing android.jar (which only contains the raw unimplemented Android API surface) and dynamically wires up enough implementation for classes and methods to be able to run a part of the Android experience on desktop. Key APIs are re-implemented using standard JRE libraries, a good example being the Android Canvas API that is translated to equivalent AWT calls for instance.

This process is done dynamically behind the scenes when you instantiate the library. It will automatically fetch the android.jar archive corresponding to the API level you want to use and plug it with the desktop implementation that is contained in another archive in the platform (aptly named layoutlib.jar ). If you are curious, you can also view that code online.

The end result is a Java ClassLoader instance that allow you to retrieve and instantiate all of the Android framework classes like android.view.View. This ClassLoader isolation is actually very handy because it allows to load several version of the platform at the same time (i.e. different android.view.View definitions) without polluting the main process class space.

In addition to this, layoutlib uses a system of callback to let consumers of the library provide additional input. One of such callback is invoked when layoutlib tries to inflate an item in a layout that it doesn’t know about because it doesn’t come from the framework ClassLoader I described above. As you probably guessed, those items are the custom user views.

Tying in Custom Controls

Because Xamarin.Android lets you use both Java and .NET based components (the former being wrapped into bindings, including our Android support libraries components), it was essential to support both of them in the designer.

Java components are actually fairly easy since the rendering process is already running in a JVM. You simply have to reference the custom component .jar code (usually distributed as part of an Android library project in a .aar file) into something like a URLClassLoader instance and wire the loading in the layoutlib callback I mentioned.

The fun stuff is obviously to include the managed components. But before we dwelve into that too much, let’s make a quick recap on how Xamarin.Android works or, more precisely, how it integrates with the Java virtual machine it’s running on.

Putting some C# in your Java

What I am going to describe here mostly relates to our legacy implementation in Xamarin.Android although much of the same ideas still live on in our new Java.Interop library (essentially in a cleaned up way).

The goal of Xamarin.Android in fine is to allow managed code to be run inside an existing Java runtime and ensure both of them can talk to each other. To accomplish this, it embeds the Mono runtime inside the process and allows it to coexist with the existing Java VM.

To allow communication between the two worlds, it uses the concept of bridged objects. Everytime a Java object is surfaced to the managed world (e.g. a overriden method parameter) or when a managed (likely autogenerated) wrapper to a Java object is created, a peering process happens whereby the managed instance and the Java instance become linked to each other, sharing their lifetime (meaning one can’t be collected without the other) until the link is cut.

To call Java code from the managed world, the JNI invocation API is used. To let Java code call into the managed world (due to subclassing or callbacks), a Java wrapper class is automatically generated.

Those wrappers (called ACW/JCW for Android/Java Callable Wrappers) contain a bunch of native methods definition to act as thunks. The thunks are then dynamically plugged using the RegisterNatives JNI call with a function pointer into essentially a mono_runtime_invoke call.

For instance, take the following MyCustomView subclass written in C# like this with its OnDraw method overriden:

public class MyCustomView: Android.Views.View
{
    public override OnDraw (Canvas c)
    {}
}

During build, an equivalent Java file will be generated, compiled and packaged in your APK:

package md55d31ab91effba0f9ed7ec79c59c38391;

public class MyCustomView
       extends android.view.View
       implements mono.android.IGCUserPeer
{
        public void onDraw (android.graphics.Canvas p0)
        {
                n_onDraw (p0);
        }

        private native void n_onDraw (android.graphics.Canvas p0);
}

When you reference that custom view in your XML layout you will likely do it in this fashion:

<MyNamespace.MyCustomView
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="match_parent"
    android:layout_height="match_parent" />

With some extra Xamarin.Android processing, this will tell the Android layout inflater to load the Java class previously generated and, when trying to draw the view, will end up calling the managed implementation of the OnDraw method through its native thunk.

Back to custom controls

Now that we have a clearer picture on how managed C# views are instantiated in an Android environment, it’s very easy to get them loaded into the designer. Simply add their generated ACW to the special URLClassLoader I mentioned beforehand and they will get processed like any other standard Java custom views.

Only thing left to do was to port our Xamarin.Android/Mono runtime to be able to run natively inside a standard Java desktop VM (in the same fashion than layoutlib with the Android API) so that it could execute the other side of that shim custom view. That code is also now open-source and you can view it on GitHub.

As part of this porting work some dusting off of our AppDomain implementation was required, especially to ensure it worked reliably in a “mobile” context. AppDomains are used as a natural complement to Java’s class loaders allowing the same silo’ed approach on both the Java and managed side.

Thanks to this association, the designer can create several context where multiple version of Android can be run together in an isolated fashion and refreshed binaries can be reloaded without the need for the renderer process itself to be torn down.

Finishing up

The layout rendered above extracted from one of my demo project contains a lot of different type of custom controls like AppBarLayout, Toolbar, RecyclerView, CoordinatorLayout and custom views for FloatingActionButton. By the way, the same feature is what’s also used in our Xamarin.Forms previewer to display the Android rendering of a layout.

There are more things that could be covered here, specifically I never mentioned how we handle error cases or even critical runtime failures which are bound to happen when you load external user code anywhere. Suffice to say we need some good error handling and process resurrection mechanisms which were definitely some of the hardest aspects to implement in the project (and still are).

Not all the code for what’s here is open-source but there are a quite few more things to look at if you are interested:

Some (old but still relevant) presentation of Xamarin.Android internals at NDC Oslo by @shana
The Java.Interop repository is a good look at how Xamarin.Android plumbing work (also has documention in eponymous folder).
Xamarin developer portal has a lot of content
- Working With JNI
- Under The Hood guides
The Xamarin.Android open-source repository