As those who read our previous article likely know, OpenFDE is a Linux desktop environment built upon the AOSP graphics stack. While it is essentially an Android-based desktop, we have adapted the display and control mechanisms for Linux applications within the AOSP environment. This enables the seamless, integrated display of both Linux and Android applications, allowing users to run programs from different platforms—and leverage their respective software ecosystems—without even noticing the underlying differences. The result is OpenFDE: a Linux desktop environment that integrates seamlessly with Android.
Of course, this technology is not limited to the OpenFDE desktop; it can also be applied to other Android systems, such as smartphones and tablets. Let’s take a look at the underlying principles.
Explanation of Principles
Our concept for OpenFDE involves providing an entry point to launch Linux applications directly; therefore, after careful consideration, we initially adopted VNC technology to handle Linux input and output management.
VNC (Virtual Network Computing) is essentially a remote desktop control technology that enables users to access and control other computers remotely via the Internet or a local area network (LAN). It works by running a VNC server program on the remote computer; the user connects to this server using a VNC client program, allowing them to view and interact with the remote computer's desktop from their own machine.
The core principle involves the VNC app communicating with the VNC server via the RFB protocol; mouse and keyboard actions are transmitted to the VNC server as input. The VNC server utilizes the Xvnc module to implement the X protocol and communicate with Linux applications, while simultaneously forwarding text (in both Chinese and English) generated by the app to the IBus input method framework for processing (the specifics of the input method mechanism will be detailed later). Keyboard and mouse inputs are passed directly to the Linux application, which returns display data to the VNC server; finally, the display frames are sent back to the app as output.
It is worth noting that VNC relies on the RFB (Remote Frame Buffer) protocol for screen sharing between the client and server. It operates by rendering images locally and then transmitting them remotely to the client for decoding and display; this architecture ensures platform independence and cross-platform compatibility.
We selected this solution due to the technology's maturity and short development cycle, which allow for rapid requirement implementation and a product-grade user experience. However, there are drawbacks: VNC is primarily designed for remote control and imposes specific demands regarding efficiency and bandwidth (though latency remains within a few frames when running locally), and there are certain limitations regarding user interaction.
Let me now explain the key points in detail.
VNC server and client
Resulting Effect
When using the TigerVNC server, the actual parameters used to start vnc server after installation via apt are as follows:
/usr/bin/Xtigervnc :2 -desktop firefox_web_browser -auth /home/warlice/.Xauthority -geometry 1900x1200 -depth 24 -rfbwait 30000 -rfbport 5902 -pn -SecurityTypes None -BlacklistThreshold=10000000 -BlacklistTimeout=0
firefox_web_browser The script content is as follows:
#!/bin/bash
fde-set-ime-engine firefox_web_browser & # Start input method, explained later
export GDK_BACKEND=x11
export QT_QPA_PLATFORM=xcb
export GTK_IM_MODULE=ibus
export QT_IM_MODULE=ibus
export QT4_IM_MODULE=ibus
export im=ibus
firefox &
The details regarding the input method are described later; here, the startup parameter configuration for vncserver specifies the layout dimensions (width and height), port, and connection timeout. Additionally, it specifies the startup program—Firefox—which is the application the user intends to launch and use. On OpenFDE, the implementation logic is as follows:
- List all available programs on the Linux system and provide launch entry points.
- When the user clicks the Linux program icon:
First, start the vncserver using the parameters mentioned above and launch the corresponding Linux application. Once startup is complete, open a new window within the FDE Android application and establish a VNC connection to display the frame data transmitted via the RFB protocol, while simultaneously sending input events—such as keyboard and mouse actions—to the server in accordance with the protocol.
This provides a seamless user experience, just like launching a Linux program directly on Android.
Source Code Solution
When chaining these functions together, modifications inevitably become necessary, requiring customization. After extensive comparative research, the project source code used is as follows.
This project involves the source code for bVNC, aRDP, aSPICE, and Opaque—four Android remote desktop clients. We are using the bVNC module; although there is a paid "Pro" version available, the documentation indicates that it merely adds encryption capabilities.
Server: GitHub - TigerVNC/tigervnc: High performance, multi-platform VNC client and server
Source code handling on the server side is somewhat more complex. The server must manage Linux applications and handle actual input and output—specifically, the graphical interfaces of these Linux programs. It utilizes the Xorg X server; VNC Server invokes Xorg via platform-specific libraries (such as Xvnc) and, following its own internal logic, converts the data into the RFB protocol for interaction with the client.
When compiling libraries on Linux that depend on xorg-xserver, you can compile x0vncserver if you only need to perform debugging.
x0vncserver - an inefficient VNC server which continuously polls any X
He cannot use all vncserver startup parameters; the result of launching it is a desktop output. You can refer toGitHub - TigerVNC/tigervnc: High performance, multi-platform VNC client and server to explore the implementation logic of the server.
Compiling TigerVNC Server
FDE uses the apt source code to modify and compile the TigerVNCServer deb package. The specific steps are as follows:
# Open /etc/apt/sources.list and uncomment all deb-src entries.
sudo apt update
sudo apt source tigervnc-standalone-server
// After entering the source code directory
sudo apt install equivs devscripts --no-install-recommends
sudo mk-build-deps -i -t "apt-get" -r
sudo DEB_BUILD_OPTIONS="parallel=8" dpkg-buildpackage -b -uc -us
The source code is managed using the quilt tool. For specific usage, please refer to:
[quilt Tool Usage Guide] f. quilt Tool Usage Guide
After successful compilation:
sudo dpkg -i tigervnc-standalone-server_1.10.1+dfsg-3_arm64.deb
Input Processing and Input Methods
After starting the server and client and establishing a connection, communication follows the RFB protocol. The output is frame data, and what frame data to display is determined by the server side, which basically does not need to be modified. Most of the client's business logic is focused on handling input, i.e., mouse and keyboard events. The simplest approach is to pass all events directly to the server side in the activity's dispatchKeyEvent, which is how bVNC handles it.
@Override
public boolean dispatchKeyEvent(KeyEvent event) {
return canvas.getKeyboard().keyEvent(event.getKeyCode(), event);
}
Send data packets according to the message format:
Protocol message - mouse event message is as follows
+--------------+--------------+--------------+
| No. of bytes | Type [Value] | Description |
+--------------+--------------+--------------+
| 1 | U8 [5] | message-type |
| 1 | U8 | button-mask |
| 2 | U16 | x-position |
| 2 | U16 | y-position |
+--------------+--------------+--------------+
Protocol message - keyboard event message is as follows
+--------------+--------------+--------------+
| No. of bytes | Type [Value] | Description |
+--------------+--------------+--------------+
| 1 | U8 [4] | message-type |
| 1 | U8 | down-flag |
| 2 | | padding |
| 4 | U32 | key |
+--------------+--------------+--------------+
For details, please refer to
[RFB Protocol Documentation] Input Protocol · RFB Remote Framebuffer Protocol
It should be noted whether the keyboard keysym used by Android is consistent with the keyboard layout recognized by Linux, that is, whether the keysym of sent events can be correctly recognized on the server side.
Input Method Requirements
In the use of VNC, there is one requirement that has never been well resolved: text input. English characters are relatively easy to handle—you can directly use key presses to output the desired characters. However, for Chinese characters and other characters in the unicode character set, Linux cannot directly generate them. It has been verified that Linux can receive a small number of Chinese characters in xkeysyms; you can bind a Chinese character to a keyboard key, but the number is also limited by the number of keyboard keys.
Therefore, if you want to input Chinese, the best approach is to add /usr/bin/ibus-daemon -d & to the startup script and then set the input method to the desired one. However, this also has a problem: the input method running on Linux is disconnected from the FDE desktop usage. How to handle the installation and uninstallation of input methods on Linux? Additionally, the display of the candidate word box is also an issue—if compatibility is not handled well, it will not be displayed in VNC.
For these reasons, FDE's VNC application integration solution adopts a more complete implementation. Modifications have been made to each stage of the input chain, though it ends at the ibus input method; X programs only serve as the text-receiving party.
The result is that using the Android input method on the Android VNC client side allows Chinese characters to be output to Linux programs opened on the server side.
This involves modifications in three stages:
The VNC APP can invoke the input method, intercept and forward characters generated by the input method to the vncserver, without affecting the original normal pathway;
The VNC server can forward received characters to the ibus input method, without affecting the original normal pathway;
The ibus input method can be normally invoked, receive character input into X programs, and does not affect the original normal input.
VNC APP
To implement input method invocation in the APP, EditText.request() suffices. However, to intercept and obtain the input method content, you need to override EditText, inject a custom InputConnection, and then obtain the input text through the commitText interface.
@Override
public boolean commitText(CharSequence text, int newCursorPosition) {
if(DEBUG){
Log.d(TAG, "commitText() called with: text = [" + text + "], mTextView = [" + detectEventEditText + "]");
}
if (detectEventEditText == null) {
return super.commitText(text, newCursorPosition);
}
if (text instanceof Spanned) {
Spanned spanned = (Spanned) text;
SuggestionSpan[] spans = spanned.getSpans(0, text.length(), SuggestionSpan.class);
Reflector.invokeMethodExceptionSafe(mIMM, "registerSuggestionSpansForNotification",
new Reflector.TypedObject(spans, SuggestionSpan[].class));
}
Reflector.invokeMethodExceptionSafe(detectEventEditText, "resetErrorChangedFlag");
Reflector.invokeMethodExceptionSafe(detectEventEditText, "hideErrorIfUnchanged");
if(mInputModeFlag == INPUT_MODE_ONLY_KEYBOARD){
return false;
}
if(!TextUtils.isEmpty(text)){
detectEventEditText.getCanvas().getKeyboard().keyEvent(0xff, null, text.toString());
}
return true;
}
Actually, you could directly use com.android.internal.inputmethod.EditableInputConnection, but it is an internal final class. The author integrated it into the APP by copying the source code, and used reflection for some functions that cannot be called directly.
Key events not generated by the input method are sent directly via the RFB protocol by calling the interface in the activity.
@Override
public boolean dispatchKeyEvent(KeyEvent event) {
return canvas.getKeyboard().keyEvent(event.getKeyCode(), event);
}
The difference is that characters generated by the input method use 0xff as the keycode, and are handled separately in RfbProto.
// Normal key press
public synchronized void writeKeyEvent(int keySym, int metaState, boolean down) {
if (viewOnly)
return;
eventBufLen = 0;
if (down) {
writeModifierKeyEvents(metaState, down);
}
if (keySym > 0)
writeKeyEvent(keySym, down);
// Always release all modifiers after an "up" event
if (!down) {
writeModifierKeyEvents(metaState, down);
}
try {
os.write(eventBuf, 0, eventBufLen);
} catch (IOException e) {
Log.e(TAG, "Failed to write key event to VNC server.");
e.printStackTrace();
}
}
// Input character
public synchronized void writeKeyStringEvent(int keycode, char c, boolean down) {
if (viewOnly || os == null )
return;
int keySym = c;
if (keySym > 0xff) {
keySym += 0x1000000; //out of keyboard
}
Log.d(TAG, "writeKeyStringEvent() called with: keySym = [" + keySym + "], down = [" + down + "]");
eventBufLen = 0;
if (keySym > 0)
writeSpacialKeyEvent(keycode, keySym, down);
try {
os.write(eventBuf, 0, eventBufLen);
} catch (IOException e) {
Log.e(TAG, "Failed to write key event to VNC server.");
e.printStackTrace();
}
}
See the code at:
bVNC/src/main/java/com/iiordanov/bVNC/RfbProto.java · OpenFDE/remote-desktop-clients - Gitee
VNC server
Upon receiving a KeyEvent from the VNC APP, according to the original protocol, the reserved bits are used as a flag to distinguish between input characters and key presses.
common/rfb/SMsgReader.cxx · OpenFDE/fde_tigervncserver - Gitee
void SMsgReader::readKeyEvent()
{
// bool down = is->readU8();
// is->skip(2);
// rdr::U32 key = is->readU32();
// handler->keyEvent(key, 0, down);
bool down = is->readU8();
rdr::U32 keycode = is->readU16();
// is->skip(2); Use this reserved bit to distinguish between input characters and key presses 0xff
rdr::U32 key = is->readU32();
handler->keyEvent(key, keycode, down);
}
Other modifications on the server side use the quilt patch approach mentioned above; the main changes are all in this patch.
debian/patches/fde_patch_001.diff · OpenFDE/fde_tigervncserver - Gitee
The key points are as follows. Simply put, a 4-digit unicode is split into 4 key press events. For example, to send "我" (unicode "6211"), it would send "6", "2", "1", "1" as 4 down/up events respectively. If there is a "0", a special keysym needs to be sent; otherwise, the ibus protocol will convert it.
/*
* vncKeyboardEvent() - add X11 events for the given RFB key event
*/
void vncKeyboardEvent(KeySym keysym, unsigned xtcode, int down)
{
/* Simple case: the client has specified the key */
// xtcode = 0xff;
if (xtcode == 0xff) {
//hardcode keysym = 0xcccc , transfer IBUS_KEY_Adiaeresis maybe never used
if (down) {
vncKeysymKeyboardEvent(0xcccc, down);
int fakeSym = (keysym >> 12) & 0xf;
if(fakeSym == 0){
fakeSendKeycode(0xcccd, down, xtcode);
fakeSendKeycode(0xcccd, 0, xtcode);
}else{
fakeSendKeycode(fakeSym, down, xtcode);
fakeSendKeycode(fakeSym, 0, xtcode);
}
fakeSym = (keysym >> 8) & 0xf;
if(fakeSym == 0){
fakeSendKeycode(0xcccd, down, xtcode);
fakeSendKeycode(0xcccd, 0, xtcode);
}else{
fakeSendKeycode(fakeSym, down, xtcode);
fakeSendKeycode(fakeSym, 0, xtcode);
}
fakeSym = (keysym >> 4) & 0xf;
if(fakeSym == 0){
fakeSendKeycode(0xcccd, down, xtcode);
fakeSendKeycode(0xcccd, 0, xtcode);
}else{
fakeSendKeycode(fakeSym, down, xtcode);
fakeSendKeycode(fakeSym, 0, xtcode);
}
fakeSym = keysym & 0xf;
if(fakeSym == 0){
fakeSendKeycode(0xcccd, down, xtcode);
fakeSendKeycode(0xcccd, 0, xtcode);
}else{
fakeSendKeycode(fakeSym, down, xtcode);
fakeSendKeycode(fakeSym, 0, xtcode);
}
} else {
vncKeysymKeyboardEvent(0xcccc, down);
}
} else{
fakeSendKeycode(keysym, down, xtcode);
}
}
This can be understood as an encoding process. The reason this treatment is necessary is due to the ibus protocol transmission process: it seems that keysym types are only not escaped when they are 1 byte, or some defined in X11's keysymdef.h.
Ibus Input Method
Ultimately, text input still uses the input method on Linux. It is started via environment variables and serves no other purpose than to decode the content sent by vncserver and generate characters.
Simply follow the ibus input method flow and register a key callback:
g_signal_connect(engine, "process-key-event", G_CALLBACK(engine_process_key_event_cb), NULL);
The keyval, keycode, and state sent by vncserver across 4 events are combined to generate a unicode character, which is then committed as text. Normal key presses do not have the flag bit 0xcccc used above, so they effectively bypass this input method.
gboolean engine_process_key_event_cb(IBusEngine *engine,
guint keyval,
guint keycode,
guint state) {
LOG_INFO("engine_process_key_event");
ibus_engine_show_lookup_table(engine);
// ibus_engine_show_preedit_text(engine);
ibus_engine_show_auxiliary_text(engine);
std::thread::id t1_id = std::this_thread::get_id();
// char receiver[100]; //
// sprintf(receiver, "\n\r Received data keycode:%d keyval:%d flag:%d text:%d ",keycode, keyval, flag, text);
// engine_commit_text(engine, ibus_text_new_from_string(receiver));
//1.unicode start
if (keyval == 0xcccc && !(state & IBUS_RELEASE_MASK)) {
// sprintf(receiver, "\n\r 1.unicode start keycode:%d maks:%d ",keycode, state & IBUS_RELEASE_MASK);
// engine_commit_text(engine, ibus_text_new_from_string(receiver));
flag = 3;
text = 0;
processing = 1;
return TRUE;
}
//2.add unicode
if (flag != -1 && !(state & IBUS_RELEASE_MASK)) {
// sprintf(receiver, "\n 2.add unicode flag:%d maks:%d ",flag, state & IBUS_RELEASE_MASK);
// engine_commit_text(engine, ibus_text_new_from_string(receiver));
if(keyval == 0xcccd){
text += (0 << (flag * 4));
}else{
text += (keyval << (flag * 4));
}
// char input_string[100]; //1
// sprintf(input_string, "\n\r Generate character bit flag:%d keyval:%d text:%d",flag, keyval, text);
// engine_commit_text(engine, ibus_text_new_from_string(input_string));
flag--;
return TRUE;
}
//3.unicode over
if (keyval == 0xcccc && (state & IBUS_RELEASE_MASK)) {
// sprintf(receiver, "\n\r 3.unicode over keycode:%d maks:%d flag:%d text:%d",keycode, state & IBUS_RELEASE_MASK, flag, text);
// engine_commit_text(engine, ibus_text_new_from_string(receiver));
flag = -1;
wchar_t unicodeChar = static_cast<wchar_t>(text);
std::wstring_convert <std::codecvt_utf8_utf16<wchar_t>> converter;
std::string utf8Str = converter.to_bytes(unicodeChar);
// char input_string[100]; //
// sprintf(input_string, "\n\r Key press keyval:%d keycode:%d text:%d final====> ", keyval, keycode, text);
// engine_commit_text(engine, ibus_text_new_from_string(input_string));
engine_commit_text(engine, ibus_text_new_from_string(utf8Str.c_str()));
processing = 0;
return TRUE;
}
if (processing) {
return TRUE;
}
text = 0;
flag = -1;
// sprintf(receiver, "\n\r 4. maks:%d " , state & IBUS_RELEASE_MASK);
// engine_commit_text(engine, ibus_text_new_from_string(receiver));
if (state & IBUS_RELEASE_MASK) {
return FALSE;
}
return FALSE;
}
Clipboard
RFB has two versions of the clipboard protocol, Clipboard · RFB Remote Framebuffer Protocol
The old one does not support Chinese, so the extended clipboard pseudo-protocol needs to be used.
RFB 3.8 protocol limitation: the clipboard can only transmit the Latin-1 character set. In 2016, Cendio Ossman merged the Extended Clipboard Pseudo-Encoding into the main protocol branch, supporting transmission of the unicode character set in clipboard messages. UltraVNC/TigerVNC/RealVNC servers all support this extended protocol, while x11vnc has not yet provided support (2021/8/11).
The extended clipboard pseudo-protocol requires support from both client and server software. The message extends ServerCutText and ClientCutText, as follows:
+--------------+--------------+--------------+
| No. of bytes | Type [Value] | Description |
+--------------+--------------+--------------+
| 1 | U8 [3/6] | message-type |
| 3 | | padding |
| 4 | S32 | length |
| 4 | U32 | text-type |
| length-4 | U8 array | text |
+--------------+--------------+--------------+
The remote-desktop-clients used in this project also does not implement this, VNC clipboard for utf8 text · Issue #285 · iiordanov/remote-desktop-clients.
The author implemented text copying through the extended clipboard pseudo-protocol. Since this protocol supports multiple formats, ZlibOutStream was used to convert the copied content. See the following patch for code modifications:
In fact, it can also implement file copying, but the author did not further implement it. remote-desktop-clients does not even implement Chinese (unicode) copying; the author speculates this is because clipboard usage scenarios on mobile are too rare, whereas OpenFDE has this requirement.
In summary, most of the principles of the VNC-based Linux application integration solution for the OpenFDE desktop environment are as described above. There are many more details to handle. It has already been released as a mature product, and we are subsequently researching methods to directly use xserver to run Linux programs within the OpenFDE desktop.
Future Plans
We are currently developing a new integration solution: porting xserver to our Android desktop. This way, starting an X Server within OpenFDE to connect to Linux programs allows direct transmission of the program's display content without going through screen image forwarding, greatly simplifying the flow. It offers better performance, compatibility, and user experience. At the same time, we will strive to achieve a window system that is as close as possible to that of Linux programs. Stay tuned.
That's all for this technical sharing session. Writing isn't easy, so if you found it helpful, please follow us for more~
Top comments (0)