DEV Community: Shai Almog

Metal Default, A New Build Cloud, And A New Format

Shai Almog — Fri, 05 Jun 2026 13:58:05 +0000

This week's release post looks different on purpose. The Friday omnibus has been getting longer and longer, and that has been working against us in two ways. SEO ignores 5,000 word pages that cover twelve unrelated topics, so the actual material gets buried instead of indexed against the queries that should find it. And when a single release post covers ten things, it becomes hard to point a colleague at "that one Codename One change from a few weeks ago" without scrolling for ten minutes.

What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com.

So from this week onwards the Friday post is the short one. A quick set of headline items, a "what is coming next" list, and that is it. The specific features get their own posts over the following days, with their own slugs, their own searchable titles, and their own discussion threads. The weekly post lives at the top of the homepage as the index; the deeper posts back-link to it; and you can read whichever ones are actually relevant to your project.

Important: it seems that if developer mode is on in your device you might get an information dialog on the right side of your UI. This issue explains how you can turn it off.

If you only have thirty seconds, here is what changed this week.

Metal is the default on iOS

PR #5065 flips the ios.metal=true build hint to the default. New iOS builds now link against CAMetalLayer instead of the deprecated CAEAGLLayer. We mentioned this three weeks ago in Metal and Skins, decided to push it back by one week last week because a couple of regressions still needed work, and shipped it this week with that list at zero.

If you have not rebuilt since this commit, your next cloud build picks Metal up automatically. No hint to add, no setting to change. The build server flipped at the same time so local builds and cloud builds match.

If you need to opt out for any reason, the hint still works in reverse:

ios.metal=false

A few things worth a glance on your first Metal build: gradient fidelity (multi-stop, conic, and repeating gradients now hit the GPU directly through PR #4957), the color space (sRGB by default, flip to displayP3 via ios.metal.colorSpace if your assets are wide gamut), and anything that draws filter: blur(...) or backdrop-filter. Everything else should look unchanged. That is the point.

A specific thank you to the community testers who flipped the hint over the past three weeks, took screenshots, and filed issues against real apps. The Metal default landed in materially better shape than it would have without you.

The new Build Cloud console is now the default link

The preview of the new Build Cloud UI went up last week. The bugs you found are fixed, and as of this PR every Dashboard link on the website now points at the new console:

https://cloud.codenameone.com/console/index.html

The navigation Dashboard link in the header, the Sign Up CTA on the pricing page, and the entries on the site map all moved. Old bookmarks still work; the legacy console stays online for the time being so you can fall back to it if something is missing or wrong in the new UI. Please tell us when you hit one of those things, because the goal is to retire the legacy URL eventually.

Historical blog posts that mention the /secure/ URL in their text were left alone.

Upcoming attractions

Three deeper posts will follow this one over the next week, each one bundling several related PRs under a single theme so the index stays small. Dates are best effort.

Developer workflow (Saturday). On-device debugging on iOS and Android, and JUnit 5 tests for Codename One apps. Codename One always had on-device debugging in the technical sense; you just had to drop into Xcode or Android Studio and jump through a depressing number of hoops. The new pipeline wires JDWP through to the real device so jdb, IntelliJ, VS Code, Eclipse, or NetBeans just attaches. The JUnit half lets you write standard @Test methods against the simulator with first-class annotations for the visual configuration (@Theme, @DarkMode, @LargerText, @Orientation, @RTL). PRs #4999, #5012, #5032.
Platform APIs in the core (Monday). Four things that move from "you need a cn1lib for this" to "it is in the framework": built-in WiFi / Bonjour / USB / network-type APIs, a modern OIDC + WebAuthn passkey identity stack (ASWebAuthenticationSession on iOS, Custom Tabs on Android), share-sheet result callbacks, and a com.codename1.ai package with LlmClient for OpenAI / Anthropic / Gemini / Ollama plus a streaming ChatView, SpeechRecognizer / TextToSpeech, and the new ML Kit cn1libs. All four share the same scanner-driven auto-injection of Android permissions and iOS entitlements that NFC and biometrics moved to two weeks ago. PRs #5021, #5018, #5039, #5036, #5035, #5057.
Build-time codegen (Wednesday). The architectural one. A reusable bytecode AnnotationProcessor SPI in the Maven plugin, the declarative router (@Route("/path"), deep links, route guards, per-tab navigation shells) that is its first concrete consumer, then a SQLite ORM (@Entity / @Id / @Column), a JSON / XML mapper (@Mapped / @JsonProperty / @XmlElement), a component binder (@Bindable / @Bind) with field-level validation, and the build-time SVG / Lottie transcoder that emits Codename One Image subclasses for every asset in src/main/svg/ or src/main/lottie/. The grab-bag PR (#5055, driven by porting a substantial mobile client app onto Codename One as the regression fixture) lands here too because the ORM and mapping work share the porting exercise that drove it. PRs #5037, #5047, #5062, #5055, #5042, #5049, #5066.

Wrapping up

That is the new format. Short post on Friday; deeper posts during the week; every change in its own place. Please tell us how it lands.

Issue tracker is here, the discussion forum is here, and the new Build Cloud console is at /console/. The Playground, Initializr, and Skin Designer are all still where they were.

NFC, Crypto, Biometrics, And A New Build Cloud

Shai Almog — Fri, 29 May 2026 14:01:40 +0000

Last week was about defaults. This week is about device APIs moving into the framework core, a small simulator change that revolutionizes Bluetooth development, and a preview of the new Build Cloud UI we would love your feedback on. There is a handful of other things in here too — and the Metal default flip I trailed last week is in a different state than I expected, which is worth a word at the end.

What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com.

A new Build Cloud UI — preview

The single most visible change this week sits behind the Build Cloud login. The console we have been serving for years is being replaced. The new UI is live now here, alongside the current console you can still find here. We want eyes and feedback on it before we flip the default.

The whole console is written in Java 17 against the Codename One UI framework, then compiled to JavaScript via our JavaScript port and served as static assets from inside the Build Cloud. Same Form, Container, BoxLayout, Toolbar, theme.css you would write for a phone build.

This is the same playbook the Initializr, the Playground, and the Skin Designer already follow. Four non-trivial Codename One apps shipping to the browser as production tooling. If you wondered whether the JavaScript port could carry a complex application UI, this is the most direct answer we can give.

Device APIs become first-class

The bigger structural change this week is that three new APIs that used to live in cn1libs or weren't available at all are now built into the framework core: biometrics, cryptography, and NFC. The unifying idea is that you should not have to add a cn1lib to do work this fundamental. The cn1lib model is still useful for genuinely third-party functionality and for features that make less sense in the core. The existing cn1libs that we are subsuming continue to work unchanged on projects that already depend on them — but the bar for what lives in core just moved.

Biometrics — PR #4987

Touch ID, Face ID, and Android BiometricPrompt are now in com.codename1.security.Biometrics. The API uses simpler semantics compared to the original fingerprint API (that predated face scanning but didn't rename the API). You can use canAuthenticate() to gate access, then an authenticate(...) call that returns an AsyncResource, typed BiometricError codes on the failure path.

Biometrics b = Biometrics.getInstance();
if (!b.canAuthenticate()) {
    // No hardware, or no enrolled biometrics
    return;
}
b.authenticate("Unlock your account").onResult((success, err) -> {
    if (err != null) {
        BiometricError code = ((BiometricException) err).getError();
        switch (code) {
            case USER_CANCELED:  return;
            case LOCKED_OUT:     fallToPassword(); return;
            case NOT_ENROLLED:   askToEnroll(); return;
            default:             fallToPassword();
        }
    } else {
        unlock();
    }
});

On iOS this wraps LocalAuthentication.framework; on Android API 29+ it uses BiometricPrompt and on API 23-28 it keeps the legacy FingerprintManager path through a reflection adapter. The build servers and local build handle permissions and framework linking seamlessly so you don't need to do anything and don't need to add a build hint. It just works.

The Java SE simulator has a new Simulate -> Biometric Simulation submenu with an Available toggle, per-modality enrollment, and a configurable outcome for the next authenticate(...) call. So you can exercise every code branch — success, user cancel, locked-out, no-hardware — without leaving the simulator.

If you have been depending on the venerable FingerprintScanner cn1lib, it continues to work unchanged. New code should reach for com.codename1.security.Biometrics.

Cryptography — PR #4994

Routine cryptography (hashing, MAC, symmetric and asymmetric encryption, signing, JWT, OTP) is now in com.codename1.security and ships with the framework. The pure-Java algorithms (Hash, Hmac, Base32, the JWT and OTP machinery) produce identical output on every supported platform. The bits that need real keys — AES, RSA, ECDSA, SecureRandom — route through each port's native crypto provider so you get hardware-backed primitives where the device offers them.

A typical AES-GCM round-trip:

SecretKey key   = KeyGenerator.aes(256);
byte[]    nonce = SecureRandom.bytes(12);
byte[]    enc   = Cipher.aesEncrypt(Cipher.AES_GCM, key, nonce, null,
                                    "secret".getBytes("UTF-8"));
byte[]    dec   = Cipher.aesDecrypt(Cipher.AES_GCM, key, nonce, null, enc);

A SHA-256 hash:

byte[] digest = Hash.sha256("hello".getBytes("UTF-8"));
String hex    = Hash.toHex(digest);

A signed JWT:

byte[] hsKey  = KeyGenerator.hmac(256);
String token  = Jwt.signHs256(hsKey)
                   .claim("sub", "user-42")
                   .claim("exp", System.currentTimeMillis() / 1000 + 3600)
                   .compact();

Jwt parsed    = Jwt.verifyHs256(token, hsKey);   // throws on bad signature
String sub    = parsed.getClaim("sub").asString();

And a TOTP that lines up with Google Authenticator / Authy:

byte[] sharedSecret = Base32.decode("JBSWY3DPEHPK3PXP");
String code         = Otp.totp(sharedSecret);    // current 30s window
boolean ok          = Otp.verifyTotp(code, sharedSecret, /* drift */ 1);

The PR also ships a matching UI widget — com.codename1.components.OtpField — a segmented, auto-advancing OTP input with paste distribution and a completion listener, so the "enter your 6-digit code" screen is now half a dozen lines of glue:

OtpField otp = new OtpField(6);
otp.setCompleteListener(code -> {
    if (Otp.verifyTotp(code, sharedSecret, 1)) {
        proceed();
    } else {
        otp.setError("Wrong code");
    }
});
form.add(otp);

We deliberately chose conservative defaults: AES/GCM/NoPadding for new authenticated AES, RSA/ECB/OAEPWithSHA-256AndMGF1Padding for new RSA, constant-time HMAC compare, a bias-free intBelow(n) on SecureRandom. The MD5 / SHA-1 / PKCS#1 / ECB transformations are still there because real apps still need to interoperate with legacy systems, but the documentation calls them out as interop-only.

NFC — PR #4996

com.codename1.nfc is the third addition. A single Nfc entry point, an NdefMessage / NdefRecord pair with typed factories (createUri, createText, createMime, createExternal, createApplicationRecord), per-technology Tag subclasses (IsoDep, MifareClassic, MifareUltralight, NfcA, NfcB, NfcF, NfcV), and a HostCardEmulationService base class for emulating a contactless card.

Reading an NDEF URI tag — the "tap a poster" pattern:

Nfc nfc = Nfc.getInstance();
if (!nfc.canRead()) return;             // no NFC hardware / NFC disabled

nfc.readTag(new NfcReadOptions()
        .setNdefOnly(true)
        .setAlertMessage("Hold near the poster"))
   .onResult((tag, err) -> {
        if (err != null) return;
        tag.readNdef().onResult((msg, e) -> {
            if (e == null) {
                String url = msg.getFirstRecord().getUriPayload();
                Display.getInstance().execute(url);
            }
        });
   });

Exchanging APDUs with an EMV / transit card:

nfc.readTag(new NfcReadOptions()
        .setTechFilter(TagType.ISO_DEP)
        .setIsoSelectAids(myAid))
   .onResult((tag, err) -> {
        if (err != null) return;
        IsoDep iso = tag.getIsoDep();
        if (iso == null) return;
        iso.transceive(myCommandApdu).onResult((resp, e) -> {
            if (ApduResponse.isSuccess(resp)) {
                /* parse response */
            }
        });
   });

Acting as a contactless card via Host Card Emulation:

class LoyaltyCard extends HostCardEmulationService {
    public String[] getAids() { return new String[] { "F0010203040506" }; }
    public byte[] processCommand(byte[] apdu) {
        return ApduResponse.withStatus(loyaltyId.getBytes("UTF-8"),
                                       ApduResponse.swSuccess());
    }
}
Nfc.getInstance().registerHostCardEmulationService(new LoyaltyCard());

Android uses NfcAdapter foreground dispatch / reader-mode and HostApduService; both manifest entries are auto-injected by the Maven plugin and the build daemon when this class is referenced. iOS uses Core NFC (NFCNDEFReaderSession, NFCTagReaderSession) for reading and CardSession (iOS 17.4+, EU only) for HCE; the NFCReaderUsageDescription plist entry and entitlements are auto-injected by the build server and local builds (again seamless is the key). The Java SE simulator has a Simulate -> NFC menu (I feel like I'm repeating myself), that lets you tap a virtual tag, edit its NDEF payload, and fire APDUs at any registered HostCardEmulationService, so you can sit at your desk and drive every code path without a card or a reader.

On platforms that do not have NFC (desktop deploy, the JavaScript port) the base class is returned and reports the device as unsupported, so application code does not need platform if statements — always gate on canRead() and you are fine.

cn1libs can now own simulator menus — and that changes Bluetooth

PR #4988 is one of those small-looking changes that opens up a whole category of UX. The Java SE simulator now scans every jar on its classpath for META-INF/codenameone/simulator-hooks.properties and lets any cn1lib contribute its own menu items. The cn1lib does not reference any Swing types — the data file just names a name=... for the menu group and a series of itemN entries pointing at public static no-arg methods, each with an optional labelN. The simulator does the rest.

A skeletal hook file:

name=Bluetooth
namespace=bluetooth

item1=com.example.bt.sim.Hooks#toggleAdapter
label1=Toggle adapter on/off

item2=com.example.bt.sim.Hooks#addDemoPeripheral
label2=Add demo peripheral

Drop that file inside a cn1lib's javase/ module and the next time the simulator starts you get a Bluetooth menu with two items in it, each running on the CN1 EDT, with Toggle adapter on/off and Add demo peripheral doing exactly what their names say. Each entry is also callable cross-platform via CN.execute("bluetooth:item1"), which is what makes the same hook usable from a screenshot test or a scripted demo. Items without a labelN are API-only — registered with the executor but hidden from the menu — which is what test suites use to prime scripted state.

We picked the data-driven shape on purpose. We are going to rewrite the simulator UX over the coming year, and we did not want cn1libs to either depend on JMenu / JMenuItem directly or have to be recompiled when the simulator's UI shell changes. The neutral SimulatorHook record (menuName, label, Runnable) is the contract; the UI on top of it is replaceable.

Bluetooth that you can actually debug

The reason the simulator hook landed this week is that we have been working on the bluetoothle-codenameone cn1lib in parallel, and the cn1lib needs the hook to be genuinely good. The result is a Bluetooth debugging story that is materially nicer than what you get out of the box on either native platform.

The library now has two backends: a real BLE backend that talks to actual hardware (CoreBluetooth on iOS, BluetoothLeScanner / BluetoothGatt on Android, the new native desktop bridge on Java SE) and a fully in-memory simulator.

To be clear, the simulator now connects to the hardware bluetooth on your device and starts scanning for real devices. I debugged bluetooth devices from my Mac using IntelliJ/IDEA and was able to see real devices!!!

The cn1lib's simulator-hooks.properties ships with seven hooks that put the simulator in the simulator's menu bar:

Bluetooth
├── Toggle adapter on/off
├── Add demo peripheral
├── Disconnect all peripherals
├── Push demo notification
├── Clear peripherals
├── Switch backend → native BLE (real hardware)
└── Switch backend → simulator

So a typical Bluetooth iteration loop looks like this:

Open your app in the Java SE simulator. The simulator backend is on by default.
Open Bluetooth -> Add demo peripheral. Your scan picks up a fake peripheral. Step through your discovery code.
Open Bluetooth → Push demo notification. Your characteristic listener fires. Step through your handler.
Open Bluetooth → Toggle adapter on/off. Your "adapter off" branch runs. Step through it.
When you are happy with the in-simulator behaviour, open Bluetooth → Switch backend → native BLE (real hardware) and your laptop's actual Bluetooth radio takes over. Same app, same code, real peripherals.

Compare that to the conventional Bluetooth iteration loop on iOS or Android. You need a real device. You need a real peripheral. The simulator does not have a BLE stack at all on iOS, and Android's emulator has a partial one that does not match real hardware. You end up doing every change on device, with cables, and the moment something goes wrong you have to figure out whether the bug is in your code, the peripheral firmware, the OS BLE stack, or some interaction between all three.

With the cn1-bluetooth simulator backend, the first four of those variables collapse to one: your code. When it works in the simulator and it does not work on device, you have narrowed the problem down to the platform BLE stack or the peripheral, which is a tractable problem. When it does not work in the simulator either, you are debugging your own code, on your own laptop.

If you have a cn1lib of your own that would benefit from a "Simulate → Whatever" menu — fake GPS coords, scripted push notifications, deterministic camera frames — the hook file is the simplest way to ship it. Two lines of properties, one public static no-arg method, and the simulator has the affordance built in.

In-app purchase consistency — PR #4990

A forum report of submitReceipt being invoked repeatedly turned into three closely related fixes in Purchase.synchronizeReceipts. All three had the same root cause: code that worked when the App Store / Play Store filled in every field, and quietly misbehaved when one of them was null.

removePendingPurchase matched only on transactionId. When a receipt's transactionId was null (a real case on some restored Android purchases) the call silently no-op'd, the receipt stayed in the pending queue, the recursion at the end of synchronizeReceipts pulled the same receipt again, and the same receipt got re-submitted forever. The fix matches on the receipt itself with a fallback tuple of (sku, storeCode, purchaseDate, orderData) when transactionId is null on either side.
The recursive synchronizeReceipts(0, callback) re-registered the caller's SuccessCallback on every iteration, so a queue of N pending receipts caused the user's callback to fire N times. The recursive call now passes null since the original callback is already in synchronizeReceiptsCallbacks.
The callback flush itself fired even when the queue had not actually drained, which masked the duplicate-submit problem at the surface and made it look like the callback was the bug.

None of this is dramatic in isolation, but the symptom — a subscription that gets re-validated against the server every few seconds — looks identical to a server bug, and it has cost real developers real hours. The fix is shipped and the regression tests cover the null-transactionId path so this exact shape does not come back.

UTF-8: JDK-compatible replace semantics + a NEON ASCII fast path — PR #4989

String.getBytes("UTF-8") and new String(bytes, "UTF-8") on iOS were behind the standard JDK in two ways. The decoder threw RuntimeException("Decoding Error") on malformed input — the rest of the Java world emits U+FFFD per maximal subpart and keeps going. The encoder dropped through to a 1-byte-per-char stub on non-Apple builds, and there was a silent ISO-8859-2 → NSISOLatin1 alias that hid encoding errors when NSString rejected the input.

The new decoder is a Hoehrmann DFA with JDK-compatible REPLACE semantics: one U+FFFD per maximal subpart violation, truncated trailing sequences also emit a U+FFFD. The encoder is a portable UTF-16 → UTF-8 with surrogate-pair joining; the Apple path now uses it directly so NSString is no longer involved in the common case. And the encoder gains a real implementation for the POSIX / test fallback in place of the old TODO stub.

The fun part is the SIMD work. The ASCII prefix scan (vmaxvq_u8) and the u8 → u16 widen (vmovl_u8) are gated on __ARM_NEON and only kick in for inputs ≥ 64 bytes. A standalone microbenchmark shows roughly 53× speedup over the scalar DFA on ASCII-heavy payloads. The integration-level benchmark cannot see this number because allocating a fresh char[] per call dominates on ParparVM, but the helpers carry their weight on the parser-style hot paths the SIMD work was added for (JSON parsing, log scanning, the kind of text that is mostly ASCII with the occasional non-ASCII codepoint).

If your app parses a lot of UTF-8 — and most apps do, because most network APIs are JSON over HTTP — this lands as a quiet but measurable speedup, and as one fewer place where iOS behaves subtly differently from the simulator.

Two long-standing JVM fixes

PR #4980 — Iterative GC mark to fix iOS stack overflow on deep graphs

Issue #3136 has been around for a long time. The ParparVM garbage collector's mark phase was recursive: for every reachable reference it followed, it pushed a stack frame, so a long linked-list chain or any deep object graph could blow the GC's own stack and crash the app. The reproducer was simple — build a LinkedList with 50000 nodes, force a GC — but the symptom on real apps was opaque: an unexplained iOS-only crash on the largest customer datasets, often weeks after the data structure was introduced.

The fix replaces the recursive mark with an iterative one over an explicit work stack. The stack lives on the heap and grows as needed, so the only ceiling now is real memory. Long linked-lists, deep trees, deeply nested JSON parsed into POJOs — all of these used to be a latent crash on iOS and now they are not.

PR #4985 — Don't rely on C arg eval order in `PUTFIELD` / `MULTIANEWARRAY`

Issue #3108 is the other one. Several PUTFIELD and MULTIANEWARRAY translation paths emitted C code that depended on argument evaluation order. C does not specify an evaluation order for function arguments. Different compilers, different optimisation levels, sometimes the same compiler at different -O levels produced different orderings, and the visible result was occasional, "miscompiled", "field was assigned the wrong value", "array dimension came out negative" bugs that nobody could reproduce reliably.

The fix is unglamorous: hoist the operand evaluations into named local variables before the storing call, so the evaluation order is fixed by the C abstract machine instead of being left to the compiler. The kind of thing where the code change is small, the testing is hard, and the symptom is "the platform feels more solid" rather than any specific feature.

I am calling these out separately from the rest because both are issues you have probably bumped into without realising it, and both are the kind of plumbing that does not show up in a feature list but quietly raises the floor under every app on iOS.

Hardware keyboard and mouse on iOS and Android — PR #4982

Issue #3498 has been on the wishlist since iPadOS started shipping with proper trackpad support and since Android pivoted to position itself as the OS Google wants on Chromebooks. The framework already exposed pointerHover* and the full keyboard event surface, but the ports did not deliver hover events at all and dropped a depressing number of hardware-keyboard keystrokes — F-keys, Esc, Tab, Home / End, PgUp / PgDn, Insert all arrived as keyPressed(0) on Android, and Enter was silently dropped unless you set sendEnterKey=true.

This PR forwards ACTION_HOVER_ENTER/MOVE/EXIT on Android into the framework's hover surface, replaces the built-in keyboard map lookup with the attached device's actual key map, includes CTRL / FN / CAPS in the meta state, and lights up the equivalent paths on iOS. Result: BT mouse, BT keyboard, stylus hover, Chromebook trackpad, iPad Magic Keyboard — all of these now do what an end user expects. Buttons highlight on hover. Tab moves focus. F-keys produce F-key codes. Cmd-C copies. Esc dismisses dialogs.

This is structural for two reasons. Android wants to replace ChromeOS for the laptop form factor, which means our Android apps are going to land on laptop-shaped devices with attached keyboards and trackpads more often than they ever have, and they need to behave like real desktop apps when they do. And iPad apps with a Magic Keyboard are increasingly indistinguishable from desktop apps in user expectation. Codename One's whole pitch is "write once, run on every screen" — the screen got a keyboard, and now we handle it.

Expanded CSS gradients and blurs — PR #4957

The CSS compiler used to reject anything past two-stop linear gradients at the four cardinal angles and two-stop radial gradients at the center, falling back to a CEF-rasterised bitmap for everything else. filter and backdrop-filter were ignored entirely. The bitmap fallback worked but it cost you the GPU path and it could not scale with the component.

This PR moves the full CSS gradient range and filter: blur(...) into native primitives end-to-end. You get multi-stop linear and radial gradients, conic gradients, repeating linear and repeating radial, the full shape and extent grammar, and Gaussian blur on both filter and backdrop-filter. Drawn on the GPU. Composable with everything else.

.HeroCard {
    background: conic-gradient(from 30deg, #ff7a00, #ff2d95, #6750a4, #ff7a00);
    border-radius: 24px;
    filter: blur(0.5px);
}

.GlassDialog {
    background: rgba(255, 255, 255, 0.18);
    backdrop-filter: blur(18px);
    border-radius: 28px;
}

The above is the kind of thing you would write today on a modern web stack. Codename One now compiles it down to the Metal / GL / Android Canvas / Swing path on the platform you are targeting, without an offscreen bitmap in the middle. Combined with the iOS Modern and Material 3 native themes we shipped three weeks ago and the accent palette overrides we shipped last week, you can put together a genuinely modern UI in pure CSS now.

On Metal: the community got there first

I said previously that I wanted to flip ios.metal=true to the default this week. That flip did not happen — and I want to be clear about why, because the reason is the best version of what we are trying to be.

The community got there first. The combination of bug reports, screenshots from real apps, and pull requests against issues people found themselves did the work of a paid QA pass. The remaining regression list is much shorter than I expected it to be a week ago. Most of the items left are subtle (specific blend modes against specific backdrops, a clip-under-rotation edge case the diagnostic test from PR #4924 has already localised, one corner case in font fallback when the device locale changes mid-session). None are showstoppers.

So instead of forcing the flip on a deadline, we are now going to flip it when the regression list reads zero. That will not be very long — within one to three weeks at the pace we are closing things — and the apps that flip first will land on a Metal default that has been tested against more real screens than any rendering migration we have done before.

If you are one of the developers who flipped the hint, took screenshots, and filed issues over the past two weeks: thank you. Keep doing it. The Metal pipeline is going to ship as the default in materially better shape than it would have without you. If you have not flipped it yet, the build hint is still ios.metal=true. We would still love your screens through it.

Wrapping up

This was a week about lifting the floor. NFC, biometrics, and cryptography are no longer optional add-ons. The simulator-hook framework opens up a class of cn1lib UX — Bluetooth being the first and largest beneficiary — that is genuinely hard to assemble on either native platform out of the box. Two of the JVM's longest-standing iOS-only bugs are finally retired. UTF-8 behaves like the standard JDK and is faster where it matters. Hardware keyboards and trackpads behave like real desktop apps would. CSS covers what a modern web stack covers.

And the Build Cloud preview is sitting on the server right now, waiting for you to break it. Please do.

A specific thank-you to the long list of community testers on the Metal pipeline (you know who you are; we are tracking the issues to a thank-you note in the next post), to Dave who submitted #3136 with the 50,000-node LinkedList repro that finally made the GC mark a one-day fix instead of a one-month investigation.

Issue tracker is here, the Playground, Initializr, and Skin Designer are all still the easiest places to see what the JavaScript port is capable of carrying. The Build Cloud preview is at /console/ on cloud.codenameone.com once you are signed in.

Skills, Java 17, And Theme Accents

Shai Almog — Fri, 22 May 2026 13:46:38 +0000

Last week was about Metal and the Skin Designer. This week the headline items are about what a brand new project looks like when you generate it: the default JDK is Java 17, and every generated project ships with an AGENTS.md authoring skill that lets any modern AI agent work on the project intelligently. There are also some other things worth covering: a runtime accent palette on the new native themes, three Metal follow-ups (one of which introduces a new matrix-correct translate API), the JDK 11+ String API gap closed, and iOS push permission that no longer fires at app launch.

What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com.

Java 17 by default

We changed the default projects generated by the Initializr to Java 17+ to focus on the future of Codename One. The existing Java 8 option in the Initializr is still selectable from the radio panel if you have a reason to use it. Pick whichever you want.

The Java 17 path is the one we now recommend for new work. Generated projects build with any JDK from 17 onwards (we routinely test on 21 and 25); you do not need to install Java 17 specifically. The bigger picture of how Java 17 support works in the toolchain, including which language features land in your app code and how the iOS / Android ports handle the newer bytecode, was covered in Official Experimental Java 17 Support earlier this year. The change this week is the default and the wording: the (Experimental) tag is gone, and Java 17 is now what you get unless you opt out.

AGENTS.md and the Codename One skill

The other change in PR #4946 is that every Java 17 project the Initializr generates now ships an AGENTS.md file at the project root and a Codename One authoring skill alongside it.

AGENTS.md is the convention for handing project-specific context to any AI agent. Claude Code, Cursor, Codex, Aider; they all look for it. Codename One projects now ship one. The actual skill content lives under .agent-skills/codename-one/ (vendor-neutral) and the source for it is in the repo at scripts/initializr/common/src/main/resources/skill if you want to read through it directly. There is also a thin stub at .claude/skills/codename-one/SKILL.md so Claude Code's /skills picker indexes it; the stub redirects to the same vendor-neutral content.

We deliberately scoped this to Java 17 projects. The older Java 8 build had additional constraints (Java 5/8 source target, retrolambda, the historical bytecode rewrite rules) that made the "what can I actually use" answer noticeably more complicated. Restricting the skill to Java 17 lets us give agents a cleaner picture of the language level, the toolchain, and the build commands without spending half the SKILL.md on caveats. If you stay on Java 8, you keep the project layout you had; nothing changes for you.

A few things the skill makes possible that I think are genuinely useful:

Agents can debug a Codename One app under jdb. This is the one I am most pleased with. The simulator is a regular JVM, so the standard Java Debugger attaches cleanly, but agents previously had no idea this workflow was available. The skill's debugging.md reference walks through starting the simulator with the right -Xrunjdwp flags, attaching jdb, setting breakpoints, dumping locals, and stepping. The same workflow works in CI and any headless context where a graphical debugger is not an option. For an LLM that is otherwise reduced to "add a println and hope", this is a much sharper tool.

Agents can check whether an API is part of the Codename One subset before they suggest it. Codename One targets a Java 5/8 shaped JDK so the same bytecode translates to iOS, Android, and JavaScript. An agent that has only read regular Java idioms will routinely reach for java.nio.file, java.time, or pieces of java.util.concurrent that the framework does not include. The skill ships a single-file IsApiSupported.java tool that an agent can invoke to verify a class or method before writing code against it.

Agents can validate a CSS snippet before applying it. Codename One CSS is its own subset; rules that look fine to a browser developer get silently dropped by the compiler. The IsCssValid.java tool lets the agent confirm the compiler will accept a snippet without booting the simulator.

These three things together are most of why an agent that was previously polite-but-not-useful on a Codename One project is now actually productive on one. If you do not use agents, the same Markdown is one of the better tours of the framework's mental model that we have written; open .agent-skills/codename-one/SKILL.md in any project you generate today and read top to bottom.

Native theme accents

PR #4884 closes the loop on the new iOS Modern and Material 3 native themes we shipped two weeks ago. The native themes now expose their accent palette as named theme constants, so rebranding your app to your own colours is a five-line CSS change instead of a fork.

Override the constants inside the #Constants block of your own theme.css:

#Constants {
    includeNativeBool: true;
    darkModeBool: true;

    --accent-color: #ff2d95;
    --accent-color-dark: #ff2d95;
    --accent-pressed-color: #c71a75;
    --accent-on-color: #ffffff;
}

That is it. Every accent-bearing UIID picks up the new colour. Light and dark are independent (--accent-color vs --accent-color-dark), and partial overrides are fine; anything you do not redeclare stays at the framework default. Material 3 has a couple of additional container-tier constants for the elevated-surface tone; iOS ignores those.

There is also a runtime path for dynamic theming (in-app accent toggles, branded flavours, A/B tests). It uses the same constants. The Native Themes chapter of the developer guide covers it in detail, along with the full iOS and Android constant tables and the places where the binding system intentionally does not apply: Accent palette override.

The point worth pulling out: the parts of theming that do not change per app (which UIIDs participate in the accent palette, which states they expose, which dark-mode counterparts they have) live inside the framework and stay there. The parts that do change per app (your colours) live in your project as five constants and nothing else. That is the whole reason this change exists.

Metal follow-ups

Last week was about shipping the Metal renderer. This week is the follow-up week: three PRs, plus one new API on Graphics that I think will quietly pay for itself many times over.

Per-axis scale decomposition (#4939, fixes #3302)

Long-standing issue #3302 had a clear repro: g.translate + g.scale(sx, sy) + fillShape with sx != sy produced shapes that visibly drifted off the axis-aligned drawRect and drawLine calls the framework emitted alongside them. Triangles inscribed in rectangles escaped their bounding rect.

The cause was that the legacy alpha-mask path rasterised the shape at a uniform scale (the diagonal ratio h2/h1), then stretched the resulting texture non-uniformly through the GPU matrix to recover the requested aspect. The bbox math is exact in real numbers, but the texture is pixel-rounded at the intermediate uniform scale, so the stretch drifted the rasterised shape off the pixel grid that drawRect and drawLine were already on.

The fix factors the user transform's 2x2 linear part by taking the column norms as (sx, sy), rasterises the path at S(sx, sy) so the per-axis stretch happens at rasterisation time against a vector path rather than a pixel grid, and applies only the residual transform * S(1/sx, 1/sy) on the GPU. The residual is pure rotation (and shear in the worst case), so no per-axis stretch happens at sample time and the alpha-mask texture lands on the same pixel grid as its drawRect siblings.

The change is gated to Metal; the GL ES2 path keeps its legacy branch so the existing GL goldens are byte-identical. A new InscribedTriangleGrid screenshot test was registered with Cn1ssDeviceRunner so the inscribed-triangle property is now visually verifiable in CI.

Clip-under-rotation diagnostic (#4924, towards #3921)

PR #4924 does not fix a bug, it localises one. Issue #3921 is "clip-under-rotation behaves wrong on some ports", entangled with a getClip / setClip(int[]) round-trip limitation the reporter himself called out as a separate issue. To split the two, we shipped a screenshot test that uses only pushClip / popClip and rotateRadians. The clip becomes non-axis-aligned via clipRect inside a 30-degree rotation, which forces the framework through its polygon-clip branch.

The expected outcome is a 30-degree-tilted red fill that overlaps the navy outline at two diagonal corners and falls short at the other two. Two distinguishable failure modes are pre-labelled in the PR: the clip widened to its axis-aligned bbox (red exactly matches the navy outline), or the polygon clip dropped entirely (red fills the whole cell). When the iOS Metal cell of this test renders, we know within a glance which of the three behaviours we are looking at. The expected-failure cell is also a hypothesis: ClipRect.m's polygon initialiser stores x = y = w = h = -1, and the Metal execute path then calls CN1MetalSetScissor(0, 0, -2, -2), whose width <= 0 / height <= 0 branch sets the scissor to the full framebuffer instead of the intended polygon. If the screenshot confirms the hypothesis, the fix is a one-line replacement of the polygon-scissor fallback.

iOS Metal colour space hint (#4909, fixes #4908)

PR #4909 adds an ios.metal.colorSpace build hint. Until this week, the Metal layer's CAMetalLayer.colorspace was hard-coded to sRGB. For most apps that is right; sRGB is what your existing assets are authored in. But on iPhone XR and later, Apple's screens are wide-gamut (Display P3), and a marketing-led brand that ships P3 artwork was visibly losing saturation by being routed through the sRGB pipeline.

Accepted values are sRGB (default), displayP3, deviceRGB, linearSRGB, extendedSRGB, extendedLinearSRGB, and none. Set it in codenameone_settings.properties:

codename1.arg.ios.metal.colorSpace=displayP3

The hint is dormant when ios.metal=false, so existing GL builds are unchanged. Unrecognised values produce a warning log and fall back to sRGB. Documented under Working-With-iOS.asciidoc.

The new `translateMatrix` API

The Inscribed-Triangle-Grid test in #4939 also surfaced a quiet papercut in Graphics that is worth pulling out as its own feature.

Graphics.translate(int, int) does not compose into the affine transform the way scale() and rotateRadians() do. It accumulates into a per-Graphics integer offset that is added to draw coordinates before the impl matrix is applied. That is a holdover from the very first version of the framework, when Graphics did not have a matrix at all. Today the consequence is surprising: a subsequent g.scale(sx, sy) multiplies the integer translate too, which means the same code produces visibly different positions depending on whether you scale before or after you translate.

The new Graphics.translateMatrix(float, float) composes the translation directly onto the impl matrix, in the same way scale and rotateRadians already do. The result is uniform "post-multiply translate onto the current transform" semantics across iOS (both GL and Metal), JavaSE, Android, and the JavaScript port. Same code, same on-screen position, whether you are drawing into a Form's Graphics or a mutable Image's Graphics.

// Matrix-correct composition. Use this when you want translate to
// behave like scale and rotate (composed into the affine transform).
g.translateMatrix(centerX, centerY);
g.rotateRadians(angle);
g.scale(sx, sy);
g.translateMatrix(-centerX, -centerY);
g.fillShape(path);

For app code writing affine-transform pipelines (the "translate to pivot, rotate, scale, translate back" idiom from Java2D and AWT), this is the API you want. isTranslateMatrixSupported() returns true on every modern port. The old translate(int, int) is not deprecated and is not going anywhere; half the framework's internal scrolling code is built on it. The new method is the one to reach for in new drawing code, particularly anything that combines translate with scale or rotate.

String API: `replace(CharSequence, CharSequence)`, `replaceAll`, `replaceFirst`

PR #4893 closes a long-standing gap reported in issue #4878. The JDK 1.5+ overload of String.replace that takes CharSequence arguments (the one nearly every modern Java tutorial reaches for) was missing from the Codename One subset. So were String.replaceAll(String, String) and String.replaceFirst(String, String). Because none of the three were on the bootclasspath, code that reached for them did not compile against a Codename One project at all; you had to know to fall back to the older replace(char, char) overload and to roll your own regex.

All three are now wired in. String.replace(CharSequence, CharSequence) has a real implementation in vm/JavaAPI. replaceAll and replaceFirst are wired through the bytecode-compliance rewriter to a new JdkApiRewriteHelper pair that delegates to the existing RE regex engine (the same pattern we have been using for years on String.split). New compliance tests cover both rewrite rules.

It is a small change in line count. In practice it is a noticeable reduction in how often "I copied a snippet from Stack Overflow and it didn't work on iOS" turns into a real bug. Three of the most-reached-for String methods in modern Java are now part of the on-device API.

iOS push permission no longer fires at app launch

PR #4894 fixes issue #4876. With ios.includePush=true the framework used to call requestAuthorizationWithOptions from application:didFinishLaunchingWithOptions:, which meant the iOS system permission dialog fired as soon as the app finished launching, before the user had seen any of your screens. There is no good way to recover from a "Don't Allow" tap at that point. The user has not experienced the app yet, does not know why notifications matter, and tapping Don't Allow is the path of least resistance. Once denied, re-prompting requires sending the user out to Settings.

The fix moves the prompt to the natural points. Push.register() triggers the system prompt (this code path already requested permission inside IOSNative.m; we just stopped firing it ahead of time). LocalNotification.schedule() also triggers it, via a new requestAuthorizationWithOptions call in sendLocalNotification. Same flow Android has been on for years. The practical consequence is that you can now show your own rationale screen ("we'd like to ping you when your order ships") before the system dialog fires.

If you have an app that needs the legacy launch-time behaviour, a backwards-compatibility build hint restores it:

codename1.arg.ios.notificationPermissionAtLaunch=true

The default is false, so existing apps that did not opt in pick up the new behaviour on next rebuild. Documented in Push-Notifications.asciidoc. The cloud-side build server change shipped as BuildDaemon #71, so local and cloud builds match.

One thing to flag if you are updating an existing iOS app: if your onboarding flow was relying on the launch-time prompt happening automatically, your prompt now never fires unless Push.register() or LocalNotification.schedule() is invoked somewhere. That is almost certainly what you want, but check that the call lands.

Skin Designer FAQ follow-up

A few questions came up on discussion #4928 after last week's Skin Designer post, worth pulling forward here because they keep coming up in the same shape:

Skins do not affect CSS. The skin is simulator scaffolding (device frame, screen rect, cutouts, safe-area insets); your theme.css and your native theme are unrelated.
For a known device, the defaults are usually right. Pick the device, hit Pick a shape, click Finish. The customization UI is there for when our device database is incomplete (the iPhone 17e entry might say "no notch" when it actually has one, or the notch position might be off by a few pixels); when you have a physical device to measure against, that is where you refine.
Themes are leaving skins. Historically the native theme was bundled inside each skin because that is what made sense at the time. Going forward the right home for themes is the framework itself, distributed via Maven, so you pick up updates automatically. The new native themes already work this way. The per-skin embedded theme stays for legacy compatibility and the Skin Designer still writes one for you, but the Native Theme menu we shipped two weeks ago is the path forward.

The device database the Skin Designer reads from is open at scripts/skindesigner/common/src/main/resources/devices.json if you want to file a PR with a device we are missing or a row whose details are off.

Wrapping up

Two reminders. First, flip ios.metal=true on your real app this week if you have not. The default flip is days away and we would rather find any remaining edge case against your screens than against the install base on launch day. Second, if you have not generated a project from the Initializr recently, do it; the Java 17 default and the AGENTS.md skill are both worth seeing for yourself.

A specific thank-you this week to the reporter on #3302 for sticking with the inscribed-triangle bug for as long as GL was the only target, Durank for the iOS push permission report on #4876, and the reporter on #4878 who flagged the missing String.replace(CharSequence, CharSequence); that one had been sitting in the gap for a long time.

Issue tracker is here, the Playground and Initializr are the easiest places to poke at the new defaults, and the Skin Designer from last week is still there if you have a device shape you need a skin for.

Metal and Skins

Shai Almog — Fri, 15 May 2026 13:34:26 +0000

This post has a lot to cover. Before we get to any of it I want to take on the uncomfortable subject first: quality. Two incidents from the past two weeks deserve a public explanation, one was a bug that fits into our normal iteration loop and one was a serious mistake on my part. Both deserve the kind of explanation I would want if I were on the other side of the import.

What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com.

How we think about quality

Codename One is a small open source company. We are not a 200-engineer platform team with a dedicated SRE rotation and a separate QA org. We move fast, fast enough that we ship meaningful new code every week, and we put a lot of effort into making sure that speed does not come at the cost of breaking your apps.

"A lot of effort" is doing some work in that sentence, so here is what it actually looks like:

Layer	Coverage
Unit and integration tests	710 Java test files exercised on every PR.
Screenshot tests	45 tests producing 190+ golden PNGs that are diffed across the iOS simulator, Android emulator, JavaSE, and headless Chrome. Both the OpenGL and Metal backends are diffed in parallel.
ParparVM bytecode-translation suite	A separate, deeper test pass that exercises VM functionality (bytecode translation, garbage collector, native interop) beyond what the framework-level tests can reach.
Cross-platform build matrix	24 GitHub workflows build every PR against iOS, Android, JavaSE, and JavaScript.
JDK matrix	JDK 8 build, JDK 11 through 25 runtime.

That is a meaningful amount of automated coverage and it catches a lot before code ever lands. What it does not catch is brand new behaviour, because there is nothing yet to compare a brand new feature against. The first golden of a new test is also the bug, until somebody actually runs the feature and tells us so.

With that in mind, let's talk about the two specific incidents from the past two weeks.

Sticky headers were half baked, and that was by design

Last week's post introduced StickyHeaderContainer with an animated transition between section headers. Within a couple of days the issue tracker had #4849, the NONE and FADE transitions were not behaving correctly and the swap had visible jitter. We turned the fix around within hours of the report.

That round trip, ship, hear from a real user, fix, is the deal we make with the community. Our pixel-diff harness is excellent at catching regressions in code that already exists. It is structurally bad at catching the first version of a brand new component because there is nothing yet to compare against. We could have sat on StickyHeaderContainer for another two weeks polishing it in private and we would have shipped a worse component, because we would not have had Thomas's eyes on it. Iterating in public, with a tight feedback loop, is how a team our size keeps moving.

The SIMD bug, which was my mistake

PR #4842 is a different story. The SIMD code on iOS uses alloca to put working buffers on the stack for speed. That is the right call for small buffers and the wrong call for large ones, past a certain size the stack request fails outright and the process crashes. The image API uses these buffers, and on large images the buffer crossed the threshold. Result: image API crash in production.

This was not a new feature. It was a change to existing, mature code. We took the precautions we always take when changing code with a long history, ran the existing tests, every existing test passed, and the patch shipped. The bug got through anyway because the test coverage validated the SIMD path on small images and never on the large ones that actually triggered the failure. That hole in the tests was mine to spot, and I did not spot it.

Once it was reported the fix turned around in well under 24 hours. The patched version detects the over-large case and falls back to a heap allocation. Small SIMD ops keep the fast alloca path. Large ones no longer crash. New tests cover the threshold case so this specific shape of bug cannot regress.

This should not have happened, but realistically it will happen again, not this exact bug, but one like it. Tests are not perfect, mine certainly are not. So the take home is the part I want to lean on:

Codename One is the first line of defense between bugs and your end users. We are not the last line. Test your application before you release. If your app supports it, use a beta channel (TestFlight on iOS, Play Console internal or closed tracks on Android) so a bug like this hits you before it hits the people who paid for your app. That tiny extra step is the most reliable protection your users have.

We are also actively brainstorming the next generation of crash protection inside the framework. The current crash protection sits at the EDT and catches RuntimeExceptions that user code throws. The next generation needs to extend further, into native crashes, into earlier startup, and into a more useful diagnostic payload that comes back to the developer instead of just the device log. There is no PR yet, we are still working out the shape, but it is the major framework-level investment we are making to give the community a stronger floor underneath their apps.

With the quality conversation out of the way, the rest of this post is about the things that actually shipped.

Metal is here

PR #4799 is the largest single change we have landed in months: a complete Metal rendering backend for iOS. It sits next to the existing OpenGL ES 2 path, behind a single build hint, with its own CI job and its own pixel-diff goldens.

Metal is Apple's modern graphics API. OpenGL on iOS was deprecated by Apple back in iOS 12, it still runs today and we kept it running for years, but "deprecated" on Apple is a slow countdown that ends with the platform pulling support. Moving to Metal now is how we get ahead of that, and it brings real benefits to your apps:

Better rendering performance. Lower draw-call overhead, modern command-encoder batching, and pipeline state caching add up to smoother scrolling and faster transitions on the same hardware.
Less battery use. Metal's reduced CPU overhead per frame means the GPU spends less time idling and the CPU spends less time bookkeeping. Long-running, graphics-heavy apps benefit the most.
Crisper text. Glyphs go through a CoreText atlas, which produces noticeably sharper rendering at the same size, with proper kerning and correct handling of complex scripts.
Pure-GPU gradients. Linear and radial gradients render entirely on the GPU instead of round-tripping through a CGContext bitmap.
Access to modern Apple graphics features. New iOS rendering features (variable-rate shading, mesh shaders, ray tracing on Apple silicon) are Metal-only. Sticking with GL means watching that train leave without us.

To enable Metal in your project, set the build hint:

ios.metal=true

Everything else stays the same. The Java surface is unchanged, your existing code keeps working.

We plan to flip Metal to be the default within two weeks, assuming no major issues surface. The ios.metal hint will stay around (set it to false to opt back into GL), but new projects and the build server's default behaviour will move over. If you ship an iOS app, please set the hint now and put your real flows through it. We want regressions to surface against your real screens, not the day after the default changes.

The most user-visible improvement from the Metal port is text. Here is the ShowcaseTheme capture from the Metal screenshot suite:

And the SpanLabelTheme capture, which is the real test for body-copy rendering, multiple lines, variable widths, the kind of paragraphs that show up in real apps:

The Metal Dialog capture is also worth showing because the translucent surface composites correctly against the textured backdrop:

The end of the skin downloader

PR #4758 ships the Skin Designer as a JavaScript bundle, embedded into the website at /skindesigner/ the same way the Playground and Initializr are embedded. You can build a skin in the browser, save it, and use it in your simulator without installing anything.

This is bigger than a website convenience. It is how we get out of the skin business.

For the entire history of Codename One, "no skin for the iPhone 16 Pro Max" or "no skin for the iPad mini 7" has been a recurring complaint, and we have published skins as fast as we could. That model never scaled. Apple ships new device sizes faster than any of us want to maintain a parallel skin catalogue, and Android has effectively infinite device shapes. Today we are deprecating the skin downloader and moving to a generic browser-based authoring tool.

To be clear about what is changing:

Existing skins are not going anywhere. Every skin that ships today will continue to work, will continue to load in the simulator, and will continue to be supported. We are not removing them. If your team has a workflow built around an existing skin, that workflow keeps working.
We will stop issuing new skins. When the next iPhone or iPad ships, we will not publish an official skin for it. Anyone can build one in the new designer in minutes, and that "anyone" includes us, of course, but it also includes you.

The "no skin for X" problem is solved generically. If you are running a niche enterprise app on a less-common Android device, you no longer have to wait on us to produce a skin for it. Build it once, drop it into your team's shared assets, done.

How the wizard works

The Skin Designer turns a device specification (resolution, PPI, fonts, safe-area insets, cutouts) into a .skin file that the JavaSE simulator can load. It runs in your browser. There is nothing to install. The wizard is intentionally opinionated. It ships with a curated device catalog, generates the device frame procedurally, and writes a skin layout that matches the iPhoneTheme.res, iOS7Theme.res, and android_holo_light.res themes shipped with Codename One.

If you only want a skin and don't care how it is built, pick a device, accept the defaults, click Finish, then Download skin. The file is ready to load via Add in the simulator's Skins menu.

Stage 1, pick a device. The first step shows a card per device from the bundled catalog. The search box filters by name (it matches both the model and the brand) and the chips below narrow by form factor: All / Phones / Tablets / Foldables. Picking a device pulls in its resolution, PPI, screen size, default safe-area insets, and the iOS or Android system font names from the catalog, then seeds a sensible starting frame: notch, island, or hole presets are applied automatically based on the device's hardware. The catalog is large, the grid is capped to the most recent matches by default, type into the search field to find older or less-common devices.

Stage 2, pick a starting source. There are three ways to seed the skin's body image:

Pick a shape generates the device frame procedurally from a small preset library (rounded rect, notch, dynamic island, punch-hole, corner hole, classic home-button). The frame is rendered as a dark gradient with the screen rect (and any cutouts) carved into it. Best when you want a generic-looking iPhone or Android frame and don't care about exact hardware fidelity.
Upload an image opens an image picker. The wizard scales the image into the device's resolution, then carves the screen rect and cutouts on top. Use this when you have a marketing render of the specific device you are targeting.
Blank rectangle collapses the bezel and corner radius to almost nothing, drops every cutout, and turns the home indicator off. The screen fills the entire skin. Useful for desktop or web simulators where the device frame would just be visual noise.

Stage 3, the editor. The editor is split into two panes: a live preview on the left that paints the device frame, screen tint, cutouts, and home indicator, and a sidebar on the right with three tabs.

The Shape tab shows a preset grid (Rounded rect, Notch, Dynamic Island, Punch-hole, Corner hole, Classic home) and dimension fields for corner radius, bezel thickness, and a toggle for the bottom home indicator. iPhones from X onward and most modern Androids should leave the indicator on, classic devices with a hardware home button should turn it off.

The Cutouts tab lists every cutout currently on the skin. Tap a row to expand its width, height, and offset fields. The three add buttons at the bottom seed a sensible default of each type. Notch (180 x 30 viewbox px) is a physical hardware cutout drawn in the device frame above the screen rect, mirroring iPhone X / 11 / 12 / 13 hardware. Island (120 x 35) is a Dynamic Island, software-reserved space rendered as an opaque pill inside the screen rect, floating on top of the iOS status bar. Hole (28 x 28) is an Android punch-hole camera, rendered like the island. When the wizard generates the .skin, it automatically extends safePortraitTop to cover any in-screen cutouts so app content lands below the floating shape.

The Info tab is mostly read-only and shows what is about to be written into skin.properties: name, width, height, PPI, pixels-per-millimeter, and the user-editable safe-area insets. The wizard intentionally does not write smallFontSize, mediumFontSize, or largeFontSize, when those are absent the simulator auto-derives them from pixelMilliRatio, which is what you want on high-PPI screens.

Stage 4, finish and download. Clicking Finish renders the portrait skin image at the device's actual resolution with rounded corners, transparent screen, opaque cutouts, and a home indicator if enabled. It synthesises the landscape skin by 90-degree rotation, writes the skin_map.png overlays that mark the screen rectangle for the simulator's screen-position detection, bundles the appropriate native theme inside the skin zip, and writes skin.properties with the platform metadata, safe-area, PPI, and display rect. Clicking Download skin hands the file to the browser's download dialog. After the file is on disk, drop it into your simulator's skins folder (or use the Add command in the simulator's Skins menu) and your new device should appear in the picker.

A generated .skin is just a renamed zip:

Apple-iPhone-16-Pro.skin/
  skin.png            # portrait body (device frame + transparent screen + cutouts)
  skin_l.png          # 90-degree rotated portrait
  skin_map.png        # black rect = screen, white = frame, used for hit-testing
  skin_map_l.png      # rotated map
  iOS7Theme.res       # bundled native theme (or android_holo_light.res / winTheme.res)
  skin.properties     # platform metadata, safe-area, PPI, display rect

The full developer-guide chapter at Skin-Designer.asciidoc walks through every stage with annotated screenshots and documents the skin.properties keys the wizard writes (roundScreen, displayX/Y/Width/Height, safePortrait*, safeLandscape*, overrideNames, system font families, PPI, and pixel ratio).

Eating our own dog food

While we're talking about the Skin Designer, this is the right moment to point out something I think is genuinely worth highlighting. The Initializr, the Playground, and the Skin Designer are all open source Codename One apps. They are written in Java using the same Codename One UI framework you use to build your iOS and Android apps, and they are deployed to the browser through our JavaScript port.

Every interaction you have with these tools, the device picker grid, the live preview rendering the device frame and cutouts, the form-driven editor with its tabbed sidebar, the file generation that bundles a .skin zip in your browser tab, is the same Codename One code that ships in your apps. The Container, Form, BoxLayout, theming, and event-handling code is identical to what you would write for a phone build. The JavaScript port translates it into something a browser can run.

These three tools are the most direct demonstration we can give of what Codename One is capable of: real, non-trivial UIs, with state, file I/O, image generation, and complex layouts, running smoothly inside a browser tab. If you have ever wondered whether the JavaScript port is production-grade enough for a real application, the Initializr, Playground, and Skin Designer are your answer. They are also the answer to "can Codename One build apps that go beyond mobile". Same codebase, deployed to a fourth target, with no rewrite.

The source for all three lives in the same CodenameOne repository the framework itself does. If you want to see how a non-trivial Codename One app is structured, those are three good places to start reading.

iOS multi-line TextArea: Return as Done

PR #4859, driven by issue #4854, gives multi-line TextArea an opt-in flag that makes the iOS keyboard's Return key act as Done. It closes the editor and fires the Done listener instead of inserting a newline. This is the iOS Reminders-app behaviour: a growing, multi-line task-title field where Return finishes the entry.

The reason it has to be a flag is that real iOS does not expose this as a built-in primitive. Reminders implements it on a UITextView whose delegate intercepts \n in shouldChangeTextInRange:. We replicate that exactly, gated behind a client property so existing layouts are untouched:

TextArea ta = new TextArea("", 3, 30);
ta.putClientProperty("iosReturnExitsEditing", Boolean.TRUE);
ta.setDoneListener(e -> { /* Return / Done was tapped */ });

While the flag is set, the keyboard's Return key is relabelled to Done (UIReturnKeyDone). Default behaviour is unchanged: the flag defaults to off, only takes effect on multi-line TextAreas, and only intercepts an exact "\n" replacement so pasted multi-line text is unaffected.

Diagnostics for status-bar tap scroll-to-top

PR #4868, driven by issue #3589, adds three complementary diagnostics for the iOS status-bar tap path. We shipped a fix earlier (#4857) and the reporter still saw no scroll on device. Rather than another sweep in the dark, we built tools to make the path observable.

Simulator menu, Simulate > iOS Status Bar Tap. Synthesises the same (displayWidth/2, 0) tap that scrollViewShouldScrollToTop: dispatches, pops a dialog reporting the responder UIID, the build-hint state, and an OK / PROBLEM verdict, then actually fires pointerPressed and pointerReleased so any wired-up scroll-to-top is observable.
Device-side properties. Display.getProperty("cn1.iosStatusBarTap.count"), cn1.iosStatusBarTap.lastEpochMillis, cn1.iosStatusBarTap.lastX/Y, and cn1.iosStatusBarTap.proxyInstalled let you inspect the path on a real iPhone. Run your app on the device, tap the status bar, and read the property. That distinguishes "iOS never delivered the message" from "iOS delivered it but a CodenameOne component intercepted the tap".
Regression coverage. StatusBarTapDiagnosticScreenshotTest exercises the exact same code path through a 2x3 frame grid, with the visible counter rising and the scroll position alternating, so future regressions surface in CI.

Simulator: Dark / Light mode toggle

PR #4871 adds a Dark / Light Mode submenu under the simulator's Simulate menu with three options: Dark Mode, Light Mode, and Unsupported (the default).

Selecting an option flips Display.isDarkMode() (Boolean.TRUE / Boolean.FALSE / null) and calls refreshSkin(...) so themes that branch on @darkModeBool re-render immediately. The choice is persisted under the cn1.simulator.darkMode Preference so the simulator restarts in the mode you left it.

Combined with the Native Theme menu we shipped two weeks ago, you can now sit on a single skin and flip between iOS Modern, Material 3, iOS 7, and Holo Light, in light, dark, and unsupported, in seconds. The everyday win is being able to verify your own theme looks right in dark mode without restarting the simulator.

Heads-up: weekend backend maintenance

This weekend we will be doing some maintenance on our build backend servers. The work is mostly invisible from the outside but it touches enough of the infrastructure that you might see intermittent build issues during the window: slower-than-usual builds, the occasional retry, possibly a short period where new builds are queued.

We are doing it because the underlying backend needs to move forward, and the cost of putting that work off keeps compounding. We will keep the disruption as short as we can. If you have a hard release deadline that lands this weekend, please plan around it. Otherwise the impact should be small and you can build through it normally.

Warning: Android 16 will effectively disallow locking orientation

Thanks to Durank for flagging #4879. The Android 16 behavior changes include a meaningful change to how Android handles orientation, in short, on large-screen devices the platform will ignore an app's request to lock orientation. If your app calls Display.lockOrientation(...) or sets a fixed orientation in the Android manifest, that lock will be honoured on phones but effectively ignored on tablets and foldables once the device targets Android 16.

There is not much we can do about this on the framework side. It is a platform-level decision and there is no public opt-out for general apps. The realistic path forward is to design layouts that work in both orientations, and to test your app against both portrait and landscape on a tablet before Android 16 reaches your users. We will keep watching for any opt-in path Google publishes, but for the moment please plan accordingly.

Why the version jumped to 7.0.242

A small note on versioning: the current release is 7.0.242, not 7.0.238 as you might expect from the cadence. The gap is real and worth explaining. We made a fix to the Maven archetype that brings over the features we added in the Codename One Initializr to projects created from the command line. The change itself is straightforward, but it interacted badly with our release build automation and we had to delete several releases along the way to get the pipeline back on its feet. The version numbers we burned in the process are the visible scar. The bright side is that command-line mvn archetype:generate now produces projects that line up with what the Initializr generates, which is what we wanted all along.

Wrapping up

We closed 24 issues in the past week, a meaningful share of them direct beneficiaries of the Metal port. Old GL-only rasterisation diffs, font sizing on retina, polygon drawing artefacts, perspective transform issues, things that the Metal pipeline simply renders correctly out of the box. Migrating the rendering layer turned out to be the cleanest way to retire a long tail of small bugs at once. With the new Skin Designer landing in the same week, two long-running structural problems went from "we should fix this someday" to "this is fixed and shipping".

If you ship an iOS app, please flip ios.metal=true this week and run your real app through it. We want to find any remaining issues now, not the day we flip the default. Issue tracker is here, the Playground is the easiest place to poke at the new themes, the Skin Designer is live on the site.

A specific thank-you this week to Thomas (@ThomasH99) for the sticky-header transition report and the Picker centring follow-up, Francesco Galgani (@jsfan3) for the iOS Reminders-style Return RFE, and the reporter on #3589 for sticking with us through a multi-PR diagnosis on the status-bar tap. The "tests cannot catch everything" section above is also a "and that is why we need you" section. It works because you keep filing.

Liquid Glass, Material 3, And A Lot Of Plumbing

Shai Almog — Wed, 06 May 2026 13:08:18 +0000

It has been one of those weeks where the diff is bigger than the headline. The headline is short — Codename One now ships modern native themes: an iOS "liquid glass" look and an Android Material 3 look, bundled into the iOS and Android ports, on by default in the Playground, and selectable from a brand new menu in the simulator. The diff behind that headline is several thousand lines across the platform ports, the simulator, the GUI plumbing, and a small army of screenshot tests.

What is Codename One? Codename One is an open-source framework for building native iOS, Android, desktop, and web apps from a single Java or Kotlin codebase. Learn more at codenameone.com.

The theme behind the work is simple: Codename One should look modern out of the box on every platform we ship to, and it should feel fast. Almost everything in the past week of commits is in service of one of those two goals.

Try it right now in the Playground

The easiest way to see any of this is the Playground. The Playground now defaults to iOS Modern when the device toggle is set to iPhone and Android Material 3 when it is set to Android, in both light and dark mode. No setup, no pom.xml, no build hints — just open the page, drop in any of the standard components, and the modern look is what you get. If the past releases of Codename One looked dated to you, the Playground is where to start.

The simulator is the second-easiest place. We will get to that.

The new native themes

For most of Codename One's life the iOS native theme has been the venerable iOS 7 flat theme, and the Android native theme has been Holo Light. Both still ship — backwards compatibility has always been one of our most important goals — but they are no longer where we want a brand new app to start. We spent the bulk of this week building two new themes that target current platform aesthetics:

iOS Modern — Apple system colors (accent #007aff light / #0a84ff dark, grouped-form surfaces, the system separator palette), pill borders for tabs, an iOS-Settings-style MultiButton, CHECK_CIRCLE-style checkbox glyphs, and translucent surfaces for Dialog and TabsContainer so they read as glass-frosted on top of whatever is behind them. It is not a real UIVisualEffectView backdrop — that is a port-side primitive we have not built yet — but the look is much closer to the iOS 26 vibe than anything we have shipped before.
Android Material 3 — the Material 3 baseline tonal palette (primary #6750a4 light / #d0bcff dark, surface-container tiers, elevated containers approximated tonally because real elevation drop-shadows are still on the to-do list), plus all the Material density and padding choices — Roboto-ish proportions, a top-tab bar with the underline-by-color treatment, the standard square checkbox glyph.

Each theme covers the usual ~25 UIIDs: base (Component, Form, ContentPane, Container), typography (Label, SecondaryLabel, TertiaryLabel, SpanLabel*), buttons (Button, RaisedButton, FlatButton with .pressed and .disabled), text input, selection controls, toolbar, tabs, side menu, list, MultiButton, dialog/sheet, FAB, and all the supporting separator and popup pieces. Both themes have full light and dark coverage.

The shipping CSS sources sit in the repo at native-themes/ios-modern/theme.css and native-themes/android-material/theme.css for anyone who wants to read what each UIID is doing.

iOS Modern

This is the ShowcaseTheme capture from the new screenshot suite, run on iOS in light and dark. Same Form, same components, swap Display.setDarkMode(...) and re-resolve. The form is built like this:

Container row = new Container(BoxLayout.x());
row.add(new Button("Default"));
Button raised = new Button("Raised");
raised.setUIID("RaisedButton");
row.add(raised);
form.add(row);

TextField tf = new TextField("hello@example.com");
form.add(tf);

Container toggles = new Container(BoxLayout.x());
CheckBox cb = new CheckBox("Remember me");
cb.setSelected(true);
toggles.add(cb);
RadioButton rb = new RadioButton("Agree");
rb.setSelected(true);
toggles.add(rb);
form.add(toggles);

SpanLabel body = new SpanLabel("Body copy …");
form.add(body);

That gives you the full picture in one screen:

The Default button uses the stock Button UIID. The Raised button uses RaisedButton, which cn1-derives from Button and adds a tinted pill on top of the iOS system blue — that is the iOS Modern accent in both modes.
The TextField is a single rounded-rect surface with the iOS system gray fill, the same shape Apple uses in Settings.
CheckBox and RadioButton use the new optional @checkBoxCheckedIconInt / @radioCheckedIconInt theme constants to swap to CHECK_CIRCLE / CHECK_CIRCLE_OUTLINE glyphs — Reminders-app aesthetic on iOS while Android keeps the standard square check.
The SpanLabel body uses the theme's base font and inherits transparent backgrounds so it never paints over a translucent parent.

The full screen source is DarkLightShowcaseThemeScreenshotTest.java.

Android Material 3

Same ShowcaseTheme source on Android. The Material 3 baseline palette gives Default the primary container color and Raised the elevated-surface tone, with the dark variant flipping the relationship correctly via the dark color-role mapping. Padding and font sizing follow Material density, which you can see in how compact the same Form lays out compared to iOS.

Translucent surfaces

This is the DialogTheme capture against the screenshot suite's textured diagonal-stripe backdrop. The backdrop is intentional — it lets reviewers see whether anything that is supposed to be translucent actually is. The iOS Modern Dialog uses an rgba surface fill (0.78 alpha in light, 0.95 in dark — dark needs more opacity because bright stripes bleed through) and its DialogBody, DialogTitle, ContentPane, CommandArea sub-UIIDs are transparent so the rounded corners read cleanly. The same trick is applied to TabsContainer and the iOS MultiButton.

Runtime palette overrides

The native theme is meant to be a starting point — you can layer your own palette on top without forking the theme. Above is the PaletteOverrideTheme capture: the base is iOS Modern, but the test layers a magenta palette on top at runtime via UIManager.addThemeProps(...). RaisedButton, FlatButton, the disabled tone, and the body-copy span all pick up the override in both light and dark — the override seam works at the resource-bundle layer, exactly the same mechanism a user theme uses to override the native theme on a real app.

In the simulator

Three pieces, all live:

Themes are bundled. The simulator jar-with-dependencies includes both modern themes alongside the four legacy themes (iPhoneTheme, iOS7Theme, androidTheme, android_holo_light) at the root of the jar. The simulator can pick any one of them at runtime without touching the skin repo.
A new "Native Theme" menu. Right next to the Skins menu there is now a Native Theme menu with a radio group for the six themes plus "Auto" and "Use skin's embedded theme". Selecting one writes the simulatorNativeTheme Preference, flips the simulator-reload flag, and disposes the current window so the skin reloader kicks in with the new theme. You can sit on a single skin and flip through every native theme in seconds.
Build hints know about it. The new nativeTheme, ios.themeMode, and and.themeMode build hints are registered with the simulator's Build Hints UI on launch — labels, types, value lists, descriptions, the lot. (The legacy keys cn1.nativeTheme and cn1.androidTheme are still honored for back-compat.) Set them in the Build Hints dialog, in codenameone_settings.properties, or via -D system properties; they flow through to the device build and the simulator both.

The "Auto" choice in the Native Theme menu defers to those build hints — set ios.themeMode=modern in your project's settings and "Auto" previews iOS Modern; flip the same project to ios.themeMode=ios7 and "Auto" previews iOS 7. The explicit menu entries (iOS Modern, iOS 7, etc.) override the hints regardless. -Dcn1.forceSimulatorTheme is still honored as the highest-priority override; pick "Use skin's embedded theme" to bypass the framework theme entirely and get whatever the skin shipped with.

On devices

The opt-in is the same on iOS and Android. The platform knobs follow a single naming pattern — ios.themeMode and and.themeMode — and accept modern / liquid / auto / ios7 / flat on iOS, modern / material / auto / hololight / legacy on Android. There is a single cross-platform shortcut, nativeTheme=modern, which the iOS builder consults when ios.themeMode is unset and which the Android port reads at runtime as a default for and.themeMode. The legacy aliases cn1.androidTheme and cn1.nativeTheme are still honored for back-compat, as is and.hololight=true.

The default for an existing app stays on legacy on every platform. We do not flip a 15-year-old app's look without an opt-in. New apps generated from the initializr ship with nativeTheme=modern, ios.themeMode=modern, and and.themeMode=modern already set in codenameone_settings.properties, so a brand new project starts with the modern themes preselected. The Playground does the same, and Playground project downloads carry the same defaults into the generated codenameone_settings.properties.

The HTML5 port has the runtime support for the modern themes but does not bundle them with user apps yet — that is one of the loose ends we want to close in the next round.

Sticky headers

The other piece of look-and-feel that we want to highlight is StickyHeaderContainer, which finally has a proper home in the framework. It is the iOS-contacts-list / sectioned-material-list component: scroll past a section boundary and the previous header is replaced by the next one. New this week, the swap is animated. A directional slide moves the outgoing header up on a forward scroll and down on a reverse scroll, or you can pick a cross-fade.

Above is a six-frame sweep from the screenshot test — the user scrolls through sections A, B, C, D, E and the pinned header recolors to whichever section is currently active at the top of the viewport.

The API is small. Build the container, register sections with addSection(header, content), configure the transition style and duration, and add it to a Form:

StickyHeaderContainer sticky = new StickyHeaderContainer();
sticky.setTransitionStyle(StickyHeaderContainer.TRANSITION_SLIDE);
sticky.setTransitionDurationMillis(250);
for (char c = 'A'; c <= 'Z'; c++) {
    Label header = new Label("" + c, "StickyHeader");
    Container items = new Container(BoxLayout.y());
    for (int i = 0; i < 5; i++) {
        items.add(new Label(c + " entry " + i));
    }
    sticky.addSection(header, items);
}
form.add(BorderLayout.CENTER, sticky);

TRANSITION_SLIDE is the default. TRANSITION_FADE cross-fades the outgoing header on top of the incoming one. TRANSITION_NONE keeps the prior instantaneous swap if you want it. Issue #4807 for the original request.

How we test this

Every screenshot in this post is captured by a test that runs the app on a real iOS device, an Android emulator, and headless Chrome, then diffs each capture against a stored golden image. The diff is the test — if the rendered pixels drift, the run fails.

For animations the test grabs a series of frames over a fixed-duration transition, then composites them into a single index image. That is how the dual-appearance shots end up as one side-by-side picture per test:

…and how the sticky-header animation ends up as a six-frame strip stitched into a GIF:

If you want to read the source, the suite lives at scripts/hellocodenameone/common/src/main/java/com/codenameone/examples/hellocodenameone/tests/.

Bugs and misc features from this week

The theme work was the loudest thing this week, but plenty of other commits landed alongside it:

SIMD large-allocation fallback. The SIMD path on iOS allocates its working buffers on the stack via alloca for speed. Past a certain buffer size the stack allocation simply fails — there is not enough stack to give, and the request crashes the process. The fix detects that case and falls back to a regular heap allocation when the request is too large to live on the stack. Small SIMD ops keep the fast alloca path; large ones no longer crash.
Pluggable AnimationTime clock. Motion, Timeline, MorphAnimation, Image.animate, and Label tickers now all route through a new AnimationTime class that defaults to System.currentTimeMillis() but can be overridden. Tests can drive animations deterministically frame by frame; demos can run in slow motion or fast forward; Motion.slowMotion is no longer the only lever.
POSIX character classes for non-ASCII letters. [[:alpha:]], [[:alnum:]], [[:lower:]], and [[:upper:]] silently failed to match anything outside the basic ASCII range — Greek, Cyrillic, CJK ideographs, accented letters, vulgar fractions, currency symbols. They now match the way you would expect, with five regression tests covering the failing cases from the issue.
Fail-fast on JDK < 11. The simulator and "Run as desktop app" goals fork the JVM with --add-exports=java.desktop/com.apple.eawt=ALL-UNNAMED, which JDK 8 rejects with the unhelpful "Could not create the Java Virtual Machine". Now the Maven plugin checks the runtime JDK version on entry to cn1:run and cn1:debug and aborts with a friendly message naming the detected version, JAVA_HOME, and a pointer to Adoptium. JDK 11 through 25 is the supported runtime range for the simulator, JDK 8 stays the build-time requirement for the core framework, and JDK 8 is still fully supported at runtime for shipped desktop apps — only the simulator / "Run as desktop app" Maven goals require JDK 11+.
Sheet scrolling swipe and animation. Sheet finally drags from the bottom with a real animation instead of snapping in. Issue #4825.
Picker positioning. Picker got additional button-positioning options and a small batch of coverage tests.
Playground polish. The Playground moved every Dialog.show(...) to InteractionDialog mode so user code calling Dialog.show does not blow away the editor chrome — it renders into the layered pane instead. Error messages got a substantial overhaul. The preview-resolution syntax expanded so the Playground can pick previews from a much wider set of expressions, with a new harness keeping it honest in CI.
Deeper refreshTheme(). Form.refreshTheme() has been around forever — it re-resolves the styles on a single Form. The new thing this week is UIManager.getInstance().refreshTheme(), which snapshots the current theme props and theme constants, clears the resolved-style caches, and re-applies the lot. This is what lets the screenshot suite flip dark mode mid-suite and see fresh styles, and what lets a runtime palette override take effect immediately. Most apps will never need to call it directly — palettes typically don't change at runtime, and a Display.setDarkMode(...) call already triggers the right invalidation. It is there if you do change the palette and want the change to stick on the next paint without reloading the theme from disk.

Where this is going — and a thank-you

Last week's post was about Codename One feeling faster: corrected pixel densities, principled scroll physics, SIMD on iOS, accessibility text scaling. This week is the symbiotic other half — Codename One looking like it belongs on a 2026 phone. Both halves are the same project. There is not much point in shipping a SIMD-accelerated Base64 if the surrounding UI looks like a 2014 app, and there is not much point in shipping a glass-frosted Dialog if the scroll underneath it judders.

Neither half is finished. They are both ongoing, and they both depend on community help — bug reports, RFEs, the patient back-and-forth on issue threads where somebody describes a layout problem on an iPhone you do not own. A specific thank you to the people who drove the issues that turned into this week's commits: Thomas (@ThomasH99) filed #4781 (the original "build a liquid glass example" RFE that started this whole effort), #4807 (sticky headers), #4838 (sideways tab swipe), #4841 (the POSIX regex fix), #4819 (picker buttons), and several others; Francesco Galgani (@jsfan3) filed #4825 (sheet swipe animation) and #4824 (light + dark theme by default in initializr); @ddyer0 caught #4811 (the EDT stack overflow) and #4767 (iPad restart Form size); Lucca Biagi (@LuccaPrado) filed #4817 (form creation in IntelliJ). Several of those are RFEs you would not file unless you actually use the framework day-to-day, and that is the kind of feedback that turns into shippable work.

We are sitting at 496 open issues as of this post. That is slow but steady progress — the number is moving in the right direction week over week, and the issues that close tend to ship as features or fixes you can see, not as silent triage. If you have a problem, file it. If you have an RFE, file that too. The themes you saw above started as an RFE.

You can try the new themes today by opening the Playground, by setting nativeTheme=modern (or ios.themeMode=modern / and.themeMode=modern for finer control) in your project's codenameone_settings.properties, or by picking them from the simulator's new Native Theme menu. New projects from the initializr already have them on. The shipping resources are bundled in the iOS and Android ports as of this week.

Front End Debugging Part 3: Networking

Shai Almog — Tue, 28 Jan 2025 16:46:20 +0000

Network Debugging Powerhouse
Re-Issuing and Modifying Requests
- cURL and Postman
Throttling and Debugging Race Conditions
- Why Throttling Matters:
- How to Use
Managing State with Storage Tools
- Challenges of State Management
Analyzing Request and Response Headers
Debugging in Incognito Mode: Limitations and Best Practices
Connecting the Front-End to the Database
Final Word

Debugging network communication issues is a critical skill for any front-end developer. While tools like Wireshark provide low-level insight into network traffic, modern browsers like Chrome and Firefox offer developer tools with powerful features tailored for web development. In this post we will discuss using browser-based tools to debug network communication issues effectively. This is a far better approach than using Wireshark for the vast majority of simple cases.

As a side note, if you like the content of this and the other posts in this series check out my Debugging book that covers this subject. If you have friends that are learning to code I'd appreciate a reference to my Java Basics book. If you want to get back to Java after a while check out my Java 8 to 21 book.

Network Debugging Powerhouse

Modern browsers come equipped with developer tools that rival standalone IDE debuggers in capability and convenience. Both Chrome and Firefox have robust network monitoring features that allow developers to observe/analyze requests and responses without leaving the browser.

On the basic level, which you’re probably familiar with, these tools include:

Network monitors: View all HTTP and HTTPS requests, including their headers, payloads, and responses.
Throttling controls: Simulate slower connections to test performance and debug race conditions.
Request replay functionality: Modify and resend requests directly from the browser.

While this post focuses on debugging techniques, it's worth noting that these tools are invaluable for performance optimization as well, though that topic warrants its own discussion.

Re-Issuing and Modifying Requests

One of the most powerful debugging features is the ability to re-issue requests. Instead of switching to external tools like cURL or Postman, browsers allow us to modify and resend requests directly.

This lets us quickly test variations of a failing API call to pinpoint issues without leaving the debugging environment. It’s especially useful when we have hard to reproduce issues or deep UI hierarchies.

In Firefox we can right-click any network entry in the Firefox Developer Tools and select "Resend." This opens an editable window where we can change request parameters, such as headers, payloads, or query strings, and resend the request.

Chrome provides similar functionality, though its interface for modifying and resending requests is slightly less direct than Firefox's.

cURL and Postman

Both browsers let you copy a request as a cURL command via the context menu. This is useful for reproducing issues in the terminal or sharing with back-end developers. I use this frequently as part of creating a reproducible issue.

If you prefer Postman, you can copy request headers and payloads from the browser and paste them into Postman to replicate requests.

Throttling and Debugging Race Conditions

Network throttling is a highly underrated feature that can be a game-changer for debugging specific classes of bugs. Both Chrome and Firefox allow developers to simulate various network speeds, from 2G connections to fast 4G.

Why Throttling Matters:

Some bugs only surface when requests arrive out of their expected order. Slowing down the network can help replicate and analyze these situations. Typical examples would be race conditions and related issues.

This is also very useful for simulating real-world conditions. Many users may not have fast or reliable internet connections. Throttling helps you understand how your application behaves in these scenarios.

I use this frequently when testing loading indicators which disappear too quickly when running locally. Instead of adding sleep code into the JavaScript or server code I can simulate slow-loading assets to verify that loading spinners or placeholders appear correctly.

How to Use

In Chrome we Open Developer Tools → Network tab.

We then use the "No throttling" dropdown to select pre-configured speeds or create a custom profile.

In Firefox we have similar functionality is available under the Network Monitor.

Managing State with Storage Tools

Local storage, session storage, and indexedDB often hold data critical to reproducing bugs, especially those tied to specific user states or devices.

Challenges of State Management

Even in incognito mode, state can persist if multiple private windows are open simultaneously. Persistence across sessions is a big challenge in these situations.

Understanding the exact state of a user's local storage can provide insight into seemingly random bugs. Debugging user-specific issues is problematic without control over storage.

In Firefox the dedicated Storage tab in Developer Tools makes it easy to inspect, edit, and delete data from local storage, session storage, cookies, and indexedDB.

In Chrome the Application tab consolidates all storage options, including the ability to clear specific caches or edit entries manually.

This functionality has many powerful uses for debugging:

Inject Debug Information: Tools like these let us manually add or modify storage data to simulate edge cases or specific user conditions.
Share Local State: Users can export their local storage, cookies, or indexedDB entries, allowing developers to reproduce issues locally.
Clear Cache Strategically: Clear only the relevant entries instead of a blanket cache clear, preserving useful state for debugging.

Analyzing Request and Response Headers

Request and response headers often hold the key to understanding network issues. We can use the network monitor to inspect:

Authorization headers: Check for missing or malformed tokens.
CORS headers: Verify that the server allows requests from your domain. These are some of the most painful type of http bugs. Reviewing these headers can be a lifesaver. If requests fail with CORS errors, inspect the Access-Control-Allow-Origin header in the response.
Cache-Control headers: Ensure proper caching behavior for your resources.

These tools are especially useful when debugging missing headers: Look for required headers like Content-Type or Authorization. Debugging Authentication: Use the "Copy as cURL" feature to test API calls with modified headers directly in the terminal.

Debugging in Incognito Mode: Limitations and Best Practices

Incognito mode can help isolate issues by providing a clean slate, however it has some limitations. Multiple incognito windows share the same state, which can lead to unintentional persistence of local data.

I suggest using storage management tools to manually clear or modify local data instead of relying solely on incognito mode. Keep only one incognito window open during testing to avoid unintended state sharing.

Connecting the Front-End to the Database

The front-end is often a transition point between user interaction and back-end data processing. While this post focuses on debugging the network layer, it's important to remember that:

Network issues often manifest due to back-end problems (e.g., a database error resulting in a 500 Internal Server Error).
Front-end developers should collaborate closely with back-end engineers to trace issues across the stack.

We can use custom response headers to include diagnostic information from the back end, such as query execution time or error codes. We can leverage server logs in conjunction with front-end debugging to get a complete picture of the issue.

Final Word

Browser developer tools are indispensable for debugging network communication issues, offering features like request replay, throttling, and storage management that simplify the debugging process. By mastering these tools, front-end developers can efficiently identify and resolve issues, ensuring a smoother user experience.

With the techniques and tips outlined in this post, you'll be better equipped to tackle network debugging challenges head-on. As you grow more familiar with these tools, you'll find them invaluable not only for debugging but also for improving your development workflow.

Front End Debugging Part 2: Console.log() to the Max

Shai Almog — Tue, 26 Nov 2024 17:53:11 +0000

Understanding Front-End Logging vs. Back-End Logging
Leveraging Console Log Levels
Customizing Console Output with CSS
Stack Tracing with console.trace()
Assertions for Design-by-Contract
Printing Tables for Clearer Data Visualization
Copying Objects to the Clipboard
Inspecting with console.dir() and dirxml()
Counting Function Calls
Organizing Logs with Groups
Chrome-Specific Debugging Features
Final Word

In my previous post I talked about why Console.log() isn’t the most effective debugging tool. In this installment, we will do a bit of an about-face and discuss the ways in which Console.log() is fantastic. Let’s break down some essential concepts and practices that can make your debugging life much easier and more productive.

Understanding Front-End Logging vs. Back-End Logging

Front-end logging differs significantly from back-end logging, and understanding this distinction is crucial. Unlike back-end systems, where persistent logs are vital for monitoring and debugging, the fluid nature of front-end development introduces different challenges. When debugging backends I’d often go for tracepoints which are far superior in that setting. However the front-end with its constant need to refresh, reload, contexts switch etc. is a very different beast. In the front-end relying heavily on elaborate logging mechanisms can become cumbersome.

While tracepoints remain superior to basic print statements, the continuous testing and browser reloading in front-end workflows lessen their advantage. Moreover, features like logging to a file or structured ingestion are rarely useful in the browser, diminishing the need for a comprehensive logging framework. However, using a logger is still considered best practice over the typical Console.log for long term logging… For short term logging Console.log has some tricks up its sleeve.

Leveraging Console Log Levels

One of the hidden gems of the browser console is its support for log levels, which is a significant step up from rudimentary print statements. The console provides five levels:

• log: Standard logging

• debug: Same as log but used for debugging purposes

• info: Informative messages, often rendered like log/debug

• warn: Warnings that might need attention

• error: Errors that have occurred

While log and debug can be indistinguishable, these levels allow for a more organized and filtered debugging experience. Browsers enable filtering the output based on these levels, mirroring the capabilities of server-side logging systems and allowing you to focus on relevant messages.

Customizing Console Output with CSS

Front-end development allows for creative solutions, and logging is no exception. Using CSS styles in the console can make logs more visually distinct. By utilizing %c in a console message, you can apply custom CSS:

console.customLog = function(msg) {
    console.log("%c" + msg,"color:black;background:pink;font-family:system-ui;font-size:4rem;-webkit-text-stroke: 1px black;font-weight:bold")
}
console.customLog("Dazzle")

This approach is helpful when you need to make specific logs stand out or organize output visually. You can use multiple %c substitutions to apply various styles to different parts of a log message.

Stack Tracing with console.trace()

The console.trace() method can print a stack trace at a particular location, which can sometimes be helpful for understanding the flow of your code. However, due to JavaScript’s asynchronous behavior, stack traces aren’t always as straightforward as in back-end debugging. Still, in specific scenarios, such as synchronous code segments or event handling, it can be quite valuable.

Assertions for Design-by-Contract

Assertions in front-end code allow developers to enforce expectations and promote a “fail-fast” mentality. Using Console.assert(), you can test conditions:

console.assert(x > 0, 'x must be greater than zero');

In the browser, a failed assertion appears as an error, similar to console.error. An added benefit is that assertions can be stripped from production builds, removing any performance impact. This makes assertions a great tool for enforcing design contracts during development without compromising production efficiency.

Printing Tables for Clearer Data Visualization

When working with arrays or objects, displaying data as tables can significantly enhance readability. The console.table() method allows you to output structured data easily:

console.table(["Simple Array", "With a few elements", "in line"])

This method is especially handy when debugging arrays of objects, presenting a clear, tabular view of the data and making complex data structures much easier to understand.

Copying Objects to the Clipboard

Debugging often involves inspecting objects, and the copy(object) method allows you to copy an object’s content to the clipboard for external use. This feature is useful when you need to transfer data or analyze it outside the browser.

Inspecting with console.dir() and dirxml()

The console.dir() method provides a more detailed view of objects, showing their properties as you’d see in a debugger. This is particularly helpful for inspecting DOM elements or exploring API responses. Meanwhile, console.dirxml() allows you to view objects as XML, which can be useful when debugging HTML structures.

Counting Function Calls

Keeping track of how often a function is called or a code block is executed can be crucial. The console.count() method tracks the number of times it’s invoked, helping you verify that functions are called as expected:

function myFunction() {
    console.count('myFunction called');
}

You can reset the counter using console.countReset(). This simple tool can help you catch performance issues or confirm the correct execution flow.

Organizing Logs with Groups

To prevent log clutter, use console groups to organize related messages. console.group() starts a collapsible log section, and console.groupEnd() closes it:

console.group('My Group');
console.log('Message 1');
console.log('Message 2');
console.groupEnd();

Grouping makes it easier to navigate complex logs and keeps your console clean.

Chrome-Specific Debugging Features

Monitoring Functions: Chrome’s monitor() method logs every call to a function, showing the arguments and enabling a method-tracing experience.

Monitoring Events: Using monitorEvents(), you can log events on an element. This is useful for debugging UI interactions. For example, monitorEvents(window, 'mouseout') logs only mouseout events.

Querying Object Instances: queryObjects(Constructor) lists all objects created with a specific constructor, giving you insights into memory usage and object instantiation.

Final Word

Front-end debugging tools have come a long way. These tools provide a rich set of features that go far beyond simple console.log() statements. From log levels and CSS styling to assertions and event monitoring, mastering these techniques can transform your debugging workflow.

If you read this post as part of my series you will notice a big change in my attitude toward debugging when we reached the front-end. Front-end debugging is very different when compared to backend debugging. When debugging the backend I’m vehemently against code changes for debugging (e.g. println debugging), but on the front-end this can be a reasonable hack. The change in environment justifies it. The short lifecycle, the single user use case and the risk is smaller.

While there are many transferrable skills we pick up while debugging, it’s important to remain flexible in our attitude. Next time we will discuss networking and storage debugging on the front-end.

Front End Debugging Part 1: Not just Console Log

Shai Almog — Tue, 29 Oct 2024 17:38:36 +0000

Instant Debugging with the debugger Keyword
Triggering Debugging from the Console
DOM Breakpoints: Monitoring DOM Changes
XHR Breakpoints: Uncovering Hidden Network Calls
Simulating Environments for Debugging
Debugging Layout and Style Issues
- Final Word

As a Java developer most of my focus is on the backend side of debugging. Front-end debugging poses different challenges and has sophisticated tools of its own. Unfortunately, print based debugging has become the norm in front-end. To be fair, it makes more sense there as the cycles are different and the problem is always a single user problem. But even if you choose to use Console.log, there’s a lot of nuance to pick up there.

Instant Debugging with the `debugger` Keyword

A cool yet powerful tool in JavaScript is the debugger keyword. Instead of simply printing a stack trace, we can use this keyword to launch the debugger directly at the line of interest. That is a fantastic tool that instantly brings your attention to a bug, I often use it in my debug builds of the front-end instead of just printing an error log.

How to Use It: Place the debugger keyword within your code, particularly within error-handling methods. When the code execution hits this line, it automatically pauses, allowing you to inspect the current state, step through code, and understand what's going wrong.

Notice that while this is incredibly useful during development, we must remember to remove or conditionally exclude debugger statements in production environments. A release build should not include these calls in a production site live environment.

Triggering Debugging from the Console

Modern browsers allow you to invoke debugging directly from the console, adding an additional layer of flexibility to your debugging process.

Example: By using the debug(functionName) command in the console, you can set a breakpoint at the start of the specified function. When this function is subsequently invoked, the execution halts, sending you directly into the debugger.

function hello(name) {
    Console.log("Hello " + name)
}
debug(hello)
hello("Shai")

This is particularly useful when you want to start debugging without modifying the source code, or when you need to inspect a function that’s only defined in the global scope.

DOM Breakpoints: Monitoring DOM Changes

DOM breakpoints are an advanced feature in Chrome and Firebug (Firefox plugin) that allow you to pause execution when a specific part of the DOM is altered.

To use it we can right-click on the desired DOM element, select “Break On,” and choose the specific mutation type you are interested in (e.g., subtree modifications, attribute changes, etc.).

DOM breakpoints are extremely powerful for tracking down issues where DOM manipulation causes unexpected results, such as dynamic content loading or changes in the user interface that disrupt the intended layout or functionality. Think of them like field breakpoints we discussed in the past.

These breakpoints complement traditional line and conditional breakpoints, providing a more granular approach to debugging complex front-end issues. This is a great tool to use when the DOM is manipulated by an external dependency.

XHR Breakpoints: Uncovering Hidden Network Calls

Understanding who initiates specific network requests can be challenging, especially in large applications with multiple sources contributing to a request. XHR (XMLHttpRequest) breakpoints provide a solution to this problem.

In Chrome or Firebug, set an XHR breakpoint by specifying a substring of the URI you wish to monitor. When a request matching this pattern is made, the execution stops, allowing you to investigate the source of the request.

This tool is invaluable when dealing with dynamically generated URIs or complex flows where tracking the origin of a request is not straightforward.

Notice that you should be selective with the filters you set; leaving the filter blank will cause the breakpoint to trigger on all XHR requests, which can become overwhelming.

Simulating Environments for Debugging

Sometimes, the issues you need to debug are specific to certain environments, such as mobile devices or different geographical locations. Chrome and Firefox offer several simulation tools to help you replicate these conditions on your desktop.

Simulating User Agents: Change the browser’s user agent to mimic different devices or operating systems. This can help you identify platform-specific issues or debug server-side content delivery that varies by user agent.
Geolocation Spoofing: Modify the browser’s reported location to test locale-specific features or issues. This is particularly useful for applications that deliver region-specific content or services.
Touch and Device Orientation Emulation: Simulate touch events or change the device orientation to see how your application responds to mobile-specific interactions. This is crucial for ensuring a seamless user experience across all devices.

These are things that are normally very difficult to reproduce. E.g. touch related issues are often challenging to debug on the device. By simulating them on the desktop browser we can shorten the debug cycle and use the tooling available on the desktop.

Debugging Layout and Style Issues

CSS and HTML bugs can be particularly tricky, often requiring a detailed examination of how elements are rendered and styled.

Inspect Element: The "inspect element" tool is the cornerstone of front-end debugging, allowing you to view and manipulate the DOM and CSS in real-time. As you make changes, the page updates instantly, providing immediate feedback on your tweaks.

Addressing Specificity Issues: One common problem is CSS specificity, where a more specific selector overrides the styles you intend to apply. The inspect element view highlights overridden styles, helping you identify and resolve conflicts.

Firefox vs. Chrome: While both browsers offer robust tools, they have different approaches to organizing these features. Firefox’s interface may seem more straightforward, with fewer tabs, while Chrome organizes similar tools under various tabs, which can either streamline your workflow or add complexity, depending on your preference.

Final Word

There are many front-end tools that I want to discuss in the coming posts. I hope you picked up a couple of new debugging tricks in this first part.

Front-end debugging requires deep understanding of browser tools and JavaScript capabilities. By mastering the techniques outlined in this post—instant debugging with the debugger keyword, DOM and XHR breakpoints, environment simulation, and layout inspection—you can significantly enhance your debugging efficiency and deliver more robust, error-free web applications.

The Art of Full Stack Debugging

Shai Almog — Tue, 08 Oct 2024 18:39:51 +0000

Full Stack Development, A Shifting Definition
- The Full Stack Debugging Approach
Frontend Debugging: Tools and Techniques
- It isn't "Just Console.log"
- The Power of Developer Tools
- Tackling Code Obfuscation
Debugging Across Layers
- Isolating Issues Across the Stack
- The Importance of System-Level Debugging
- Embracing Complexity
Conclusion

Full stack development is often likened to an intricate balancing act, where developers are expected to juggle multiple responsibilities across the frontend, backend, database, and beyond. As the definition of full stack development continues to evolve, so too does the approach to debugging. Full stack debugging is an essential skill for developers, as it involves tracking issues through multiple layers of an application, often navigating domains where one’s knowledge may only be cursory. In this blog post I aim to explore the nuances of full stack debugging, offering practical tips and insights for developers navigating the complex web of modern software development.

Notice that this is an introductory post focusing mostly on the front end debugging aspects, in the following posts I will dig deeper into the less familiar capabilities in front end debugging.

Full Stack Development, A Shifting Definition

The definition of full stack development is as fluid as the technology stacks themselves. Traditionally, full stack developers were defined as those who could work on both the frontend and backend of an application. However, as the industry evolves, this definition has expanded to include aspects of operations (OPS) and configuration. The modern full stack developer is expected to submit pull requests that cover all parts required to implement a feature—backend, database, frontend, and configuration. While this does not make them an expert in all these areas, it does require them to navigate across domains, often relying on domain experts for guidance.

I've heard it said that full stack developers are:

Jack of all trades, master of none.

However, the full quote probably better represents the reality:

Jack of all trades, master of none, but better than a master of one.

The Full Stack Debugging Approach

Just as full stack development involves working across various domains, full stack debugging requires a similar approach. A symptom of a bug may manifest in the frontend, but its root cause could lie deep within the backend or database layers. Full stack debugging is about tracing these issues through the layers and isolating them as quickly as possible. This is no easy task, especially when dealing with complex systems where multiple layers interact in unexpected ways. The key to successful full stack debugging lies in understanding how to track an issue through each layer of the stack and identifying common pitfalls that developers may encounter.

Frontend Debugging: Tools and Techniques

It isn't "Just Console.log"

Frontend developers are often stereotyped as relying solely on Console.log for debugging. While this method is simple and effective for basic debugging tasks, it falls short when dealing with the complex challenges of modern web development. The complexity of frontend code has increased significantly, making advanced debugging tools not just useful, but necessary. Yet, despite the availability of powerful debugging tools, many developers continue to shy away from them, clinging to old habits.

The Power of Developer Tools

Modern web browsers come equipped with robust developer tools that offer a wide range of capabilities for debugging frontend issues. These tools, available in browsers like Chrome and Firefox, allow developers to inspect elements, view and edit HTML and CSS, monitor network activity, and much more. One of the most powerful, yet underutilized, features of these tools is the JavaScript debugger.

The debugger allows developers to set breakpoints, step through code, and inspect the state of variables at different points in the execution. However, the complexity of frontend code, particularly when it has been obfuscated for performance reasons, can make debugging a challenging task.

We can launch the browser tools on Firefox using this menu:

On Chrome we can use this option:

I prefer working with Firefox, I find their developer tools more convenient but both browsers have similar capabilities. Both have fantastic debuggers (as you can see with the Firefox debugger below), unfortunately many developers limit themselves to console printing instead of exploring this powerful tool.

Tackling Code Obfuscation

Code obfuscation is a common practice in frontend development, employed to protect proprietary code and reduce file sizes for better performance. However, obfuscation also makes the code difficult to read and debug. Fortunately, both Chrome and Firefox developer tools provide a feature to de-obfuscate code, making it more readable and easier to debug. By clicking the curly brackets button in the toolbar, developers can transform a single line of obfuscated code into a well-formed, debuggable file.

Another important tool in the fight against obfuscation is the source map. Source maps are files that map obfuscated code back to its original source code, including comments. When generated and properly configured, source maps allow developers to debug the original code instead of the obfuscated version. In Chrome, this feature can be enabled by ensuring that "Enable JavaScript source maps" is checked in the developer tools settings.

You can use code like this in the JavaScript file to point at the sourcemap file:

//@sourceMappingURL=myfile.js.map

For this to work in Chrome we need to ensure that "Enable JavaScript source maps" is checked in the settings. Last I checked it was on by default but it doesn't hurt to verify:

Debugging Across Layers

Isolating Issues Across the Stack

In full stack development, issues often manifest in one layer but originate in another. For example, a frontend error might be caused by a misconfigured backend service or a database query that returns unexpected results. Isolating the root cause of these issues requires a methodical approach, starting from the symptom and working backward through the layers.

A common strategy is to reproduce the issue in a controlled environment, such as a local development setup, where each layer of the stack can be tested individually. This helps to narrow down the potential sources of the problem. Once the issue has been isolated to a specific layer, developers can use the appropriate tools and techniques to diagnose and resolve it.

The Importance of System-Level Debugging

Full stack debugging is not limited to the application code. Often, issues arise from the surrounding environment, such as network configurations, third-party services, or hardware limitations. A classic example of this that we ran into a couple of years ago was a production problem where a WebSocket connection would frequently disconnect. After extensive debugging, Steve discovered that the issue was caused by the CDN provider (CloudFlare) timing out the WebSocket after two minutes—something that could only be identified by debugging the entire system, not just the application code.

System-level debugging requires a broad understanding of how different components of the infrastructure interact with each other. It also involves using tools that can monitor and analyze the behavior of the system as a whole, such as network analyzers, logging frameworks, and performance monitoring tools.

Embracing Complexity

Full stack debugging is inherently complex, as it requires developers to navigate multiple layers of an application, often dealing with unfamiliar technologies and tools. However, this complexity also presents an opportunity for growth. By embracing the challenges of full stack debugging, developers can expand their knowledge and become more versatile in their roles.

One of the key strengths of full stack development is the ability to collaborate with domain experts. When debugging an issue that spans multiple layers, it is important to leverage the expertise of colleagues who specialize in specific areas. This collaborative approach not only helps to resolve issues more efficiently but also fosters a culture of knowledge sharing and continuous learning within the team.

As tools continue to evolve, so too do the tools and techniques available for debugging. Developers should strive to stay up-to-date with the latest advancements in debugging tools and best practices. Whether it’s learning to use new features in browser developer tools or mastering system-level debugging techniques, continuous learning is essential for success in full stack development.

Conclusion

Full stack debugging is a critical skill for modern developers, we mistakenly think it requires deep understanding of both the application and its surrounding environment. I disagree... By mastering the tools and techniques discussed in this post/upcoming posts, developers can more effectively diagnose and resolve issues that span multiple layers of the stack. Whether you’re dealing with obfuscated frontend code, misconfigured backend services, or system-level issues, the key to successful debugging lies in a methodical, collaborative approach.

You don't need to understand every part of the system, just the ability to eliminate the impossible.

Mastering Serverless Debugging

Shai Almog — Tue, 02 Jul 2024 16:26:02 +0000

Introduction to Serverless Computing
Challenges of Serverless Debugging
- Disconnected Environments
- Lack of Standardization
- Limited Debugging Tools
- Concurrency and Scale
Effective Strategies for Serverless Debugging
- Local Debugging with IDE Remote Capabilities
- Using Feature Flags for Debugging
- Staged Rollouts and Canary Deployments
- Comprehensive Logging
- Embracing Idempotency
Debugging a Lambda Application Locally with AWS SAM
- Setting Up the Local Environment
- Running the Hello World Application Locally
- Configuring Remote Debugging
- Handling Debugger Timeouts
Final Word

Serverless computing has emerged as a transformative approach to deploying and managing applications. The theory is that by abstracting away the underlying infrastructure, developers can focus solely on writing code. While the benefits are clear—scalability, cost efficiency, and performance—debugging serverless applications presents unique challenges. This post explores effective strategies for debugging serverless applications, particularly focusing on AWS Lambda.

Before I proceed I think it's important to disclose a bias: I am personally not a huge fan of Serverless or PaaS after I was burned badly by PaaS in the past. However, some smart people like Adam swear by it so I should keep an open mind.

Introduction to Serverless Computing

Serverless computing, often referred to as Function as a Service (FaaS), allows developers to build and run applications without managing servers. In this model, cloud providers automatically handle the infrastructure, scaling, and management tasks, enabling developers to focus purely on writing and deploying code. Popular serverless platforms include AWS Lambda, Azure Functions, and Google Cloud Functions.

In contrast, Platform as a Service (PaaS) offers a more managed environment where developers can deploy applications but still need to configure and manage some aspects of the infrastructure. PaaS solutions, such as Heroku and Google App Engine, provide a higher level of abstraction than Infrastructure as a Service (IaaS) but still require some server management.

Kubernetes, which we recently discussed, is an open-source container orchestration platform that automates the deployment, scaling, and management of containerized applications. While Kubernetes offers powerful capabilities for managing complex, multi-container applications, it requires significant expertise to set up and maintain. Serverless computing simplifies this by removing the need for container orchestration and management altogether.

The "catch" is two fold:

Serverless programming removes the need to understand the servers but also removes the ability to rely on them resulting in more complex architectures.
Pricing starts off cheap. Practically free. It can quickly escalate especially in case of an attack or misconfiguration.

Challenges of Serverless Debugging

While serverless architectures offer some benefits, they also introduce unique debugging challenges. The primary issues stem from the inherent complexity and distributed nature of serverless environments. Here are some of the most pressing challenges.

Disconnected Environments

One of the major hurdles in serverless debugging is the lack of consistency between development, staging, and production environments. While traditional development practices rely on these separate environments to test and validate code changes, serverless architectures often complicate this process. The differences in configuration and scale between these environments can lead to bugs that only appear in production, making them difficult to reproduce and fix.

Lack of Standardization

The serverless ecosystem is highly fragmented, with various vendors offering different tools and frameworks. This lack of standardization can make it challenging to adopt a unified debugging approach. Each platform has its own set of practices and tools, requiring developers to learn and adapt to multiple environments.

This is slowly evolving with some platforms gaining traction, but since this is a vendor driven industry there are many edge cases.

Limited Debugging Tools

Traditional debugging tools, such as step-through debugging and breakpoints, are often unavailable in serverless environments. The managed and controlled nature of serverless functions restricts access to these tools, forcing developers to rely on alternative methods, such as logging and remote debugging.

Concurrency and Scale

Serverless functions are designed to handle high concurrency and scale seamlessly. However, this can introduce issues that are hard to reproduce in a local development environment. Bugs that manifest only under specific concurrency conditions or high load are particularly challenging to debug.

Notice that when I discuss concurrency here I'm often referring to race conditions between separate services.

Effective Strategies for Serverless Debugging

Despite these challenges, several strategies can help make serverless debugging more manageable. By leveraging a combination of local debugging, feature flags, staged rollouts, logging, idempotency, and Infrastructure as Code (IaC), developers can effectively diagnose and fix issues in serverless applications.

Local Debugging with IDE Remote Capabilities

While serverless functions run in the cloud, you can simulate their execution locally using tools like AWS SAM (Serverless Application Model). This involves setting up a local server that mimics the cloud environment, allowing you to run tests and perform basic trial-and-error debugging.

To get started, you need to install Docker or Docker Desktop, create an AWS account, and set up the AWS SAM CLI. Deploy your serverless application locally using the SAM CLI, which enables you to run the application and simulate Lambda functions on your local machine. Configure your IDE for remote debugging, launching the application in debug mode, and connecting your debugger to the local host. Set breakpoints to step through the code and identify issues.

Using Feature Flags for Debugging

Feature flags allow you to enable or disable parts of your application without deploying new code. This can be invaluable for isolating issues in a live environment. By toggling specific features on or off, you can narrow down the problematic areas and observe the application’s behavior under different configurations.

Implementing feature flags involves adding conditional checks in your code that control the execution of specific features based on the flag’s status. Monitoring the application with different flag settings helps identify the source of bugs and allows you to test fixes without affecting the entire user base.

This is essentially "debugging in production". Working on a new feature?

Wrap it in a feature flag which is effectively akin to wrapping the entire feature (client and server) in if statements. You can then enable it conditionally globally or on a per user basis. This means you can test the feature, enable or disable it based on configuration without redeploying the application.

Staged Rollouts and Canary Deployments

Deploying changes incrementally can help catch bugs before they affect all users. Staged rollouts involve gradually rolling out updates to a small percentage of users before a full deployment. This allows you to monitor the performance and error logs of the new version in a controlled manner, catching issues early.

Canary deployments take this a step further by deploying new changes to a small subset of instances (canaries) while the rest of the system runs the stable version. If issues are detected in the canaries, you can roll back the changes without impacting the majority of users. This method limits the impact of potential bugs and provides a safer way to introduce updates. This isn't great as in some cases some demographics might be more reluctant to report errors. However, for server side issues this might make sense as you can see the impact based on server logs and metrics.

Comprehensive Logging

Logging is one of the most common and essential tools for debugging serverless applications. I wrote and spoke a lot about logging in the past. By logging all relevant data points, including inputs and outputs of your functions, you can trace the flow of execution and identify where things go wrong.

However, excessive logging can increase costs, as serverless billing is often based on execution time and resources used. It’s important to strike a balance between sufficient logging and cost efficiency. Implementing log levels and selectively enabling detailed logs only when necessary can help manage costs while providing the information needed for debugging.

I talk about striking the delicate balance between debuggable code, performance and cost with logs in the following video. Notice that this is a general best practice and not specific to serverless.

Embracing Idempotency

Idempotency, a key concept from functional programming, ensures that functions produce the same result given the same inputs, regardless of the number of times they are executed. This simplifies debugging and testing by ensuring consistent and predictable behavior.

Designing your serverless functions to be idempotent involves ensuring that they do not have side effects that could alter the outcome when executed multiple times. For example, including timestamps or unique identifiers in your requests can help maintain consistency. Regularly testing your functions to verify idempotency can make it easier to pinpoint discrepancies and debug issues.

Testing is always important but in serverless and complex deployments it becomes critical. Awareness and embrace of idempotency allows for more testable code and easier to reproduce bugs.

Debugging a Lambda Application Locally with AWS SAM

Debugging serverless applications, particularly AWS Lambda functions, can be challenging due to their distributed nature and the limitations of traditional debugging tools. However, AWS SAM (Serverless Application Model) provides a way to simulate Lambda functions locally, enabling developers to test and debug their applications more effectively. I will use it as a sample to explore the process of setting up a local debugging environment, running a sample application, and configuring remote debugging.

Setting Up the Local Environment

Before diving into the debugging process, it's crucial to set up a local environment that can simulate the AWS Lambda environment. This involves a few key steps:

Install Docker: Docker is required to run the local simulation of the Lambda environment. You can download Docker or Docker Desktop from the official Docker website.
Create an AWS Account: If you don't already have an AWS account, you need to create one. Follow the instructions on the AWS account creation page.
Set Up AWS SAM CLI: The AWS SAM CLI is essential for building and running serverless applications locally. You can install it by following the AWS SAM installation guide.

Running the Hello World Application Locally

To illustrate the debugging process, let's use a simple "Hello World" application. The code for this application can be found in the AWS Hello World tutorial.

Deploy Locally: Use the SAM CLI to deploy the Hello World application locally. This can be done with the following command:
```
sam local start-api
```
This command starts a local server that simulates the AWS Lambda cloud environment.
Trigger the Endpoint: Once the local server is running, you can trigger the endpoint using a curl command:
```
curl http://localhost:3000/hello
```
This command sends a request to the local server, allowing you to test the function's response.

Configuring Remote Debugging

While running tests locally is a valuable step, it doesn't provide full debugging capabilities. To debug the application, you need to configure remote debugging. This involves several steps.

First we need to start the application in debug mode using the following SAM command:

sam local invoke -d 5858

This command pauses the application and waits for a debugger to connect.

Next we need to configure the IDE for remote debugging. We start by setting up the IDE to connect to the local host for remote debugging. This typically involves creating a new run configuration that matches the remote debugging settings.

We can now set breakpoints in the code where we want the execution to pause. This allows us to step through the code and inspect variables and application state just like in any other local application.

We can test this by invoking the endpoint e.g. using curl. With the debugger connected we would stop on the breakpoint like any other tool:

curl http://localhost:3000/hello

The application will pause at the breakpoints you set, allowing you to step through the code.

Handling Debugger Timeouts

One significant challenge when debugging Lambda functions is the quick timeout setting. Lambda functions are designed to execute quickly, and if they take too long, the costs can become prohibitive. By default, the timeout is set to a short duration, but you can configure this in the template.yaml file e.g.:

Resources:
  HelloWorldFunction:
    Type: AWS::Serverless::Function
    Properties:
      Handler: app.lambdaHandler
      Timeout: 60  # timeout in seconds

After updating the timeout value, re-issue the sam build command to apply the changes.

In some cases, running the application locally might not be enough. You may need to simulate running on the actual AWS stack to get more accurate debugging information. Solutions like SST (Serverless Stack) or MerLoc can help achieve this, though they are specific to AWS and relatively niche.

Final Word

Serverless debugging requires a combination of strategies to effectively identify and resolve issues. While traditional debugging methods may not always apply, leveraging local debugging, feature flags, staged rollouts, comprehensive logging, idempotency, and IaC can significantly improve your ability to debug serverless applications. As the serverless ecosystem continues to evolve, staying adaptable and continuously updating your debugging techniques will be key to success.

Debugging serverless applications, particularly AWS Lambda functions, can be complex due to their distributed nature and the constraints of traditional debugging tools. However, by leveraging tools like AWS SAM, you can simulate the Lambda environment locally and use remote debugging to step through your code. Adjusting timeout settings and considering advanced simulation tools can further enhance your debugging capabilities.

Debugging Kubernetes - Troubleshooting Guide

Shai Almog — Tue, 18 Jun 2024 14:26:44 +0000

Identifying Configuration Issues
- Common Causes and Solutions
- Detailed Investigation Steps
Dealing with Image Pull Errors
- Troubleshooting Steps
Handling Node Issues
- Preventive Measures
Managing Missing Configuration Keys or Secrets
Utilizing Buildg for Interactive Debugging
Conclusion

As Kubernetes continues to revolutionize the way we manage and deploy applications, understanding its intricacies becomes essential for developers and operations teams alike. If you don't have a dedicated DevOps team you probably shouldn't be working with Kubernetes. Despite that, in some cases a DevOps engineer might not be available while we're debugging an issue. For these situations and for general familiarity we should still familiarize ourselves with common Kubernetes issues to bridge the gap between development and operations. I think this also provides an important skill that helps us understand the work of DevOps better, with that understanding we can improve as a cohesive team. This guide explores prevalent Kubernetes errors and provides troubleshooting tips to help developers navigate the complex landscape of container orchestration.

Identifying Configuration Issues

When you encounter configuration issues in Kubernetes, the first place to check is the status column using the kubectl get pods command. Common errors manifest here, requiring further inspection with kubectl describe pod.

$ kubectl get pods
NAME                     READY    STATUS     RESTARTS   AGE 
my-first-pod-id-xxxx      1/1     Running    0          13s
my-second-pod-id-xxxx     1/1     Running    0          13s

Common Causes and Solutions

Insufficient Resources: Notice that this means resources for the POD itself and not resources within the container. It means the hardware or surrounding VM is hitting a limit.

Symptom: Pods fail to schedule due to resource constraints.

Solution: Scale up the cluster by adding more nodes to accommodate the resource requirements.

Volume Mounting Failures:

Symptom: Pods cannot mount volumes correctly.

Solution: Ensure storage is defined accurately in the pod specification and check the storage class and Persistent Volume (PV) configurations.

Detailed Investigation Steps

We can use kubectl describe pod: This command provides a detailed description of the pod, including events that have occurred. By examining these events, we can pinpoint the exact cause of the issue.

Another important step is resource quota analysis. Sometimes, resource constraints are due to namespace-level resource quotas. Use kubectl get resourcequotas to check if quotas are limiting pod creation.

Dealing with Image Pull Errors

Errors like ErrImagePull or ImagePullBackOff indicate issues with fetching container images. These errors are typically related to image availability or access permissions.

Troubleshooting Steps

The first step is checking the image name which we can do with the following command:

docker pull <image-name>

We then need to verify the image name for typos or invalid characters. I pipe the command through grep to verify the name is 100% identical, some typos are just notoriously hard to spot.

Credentials can also be a major pitfall. E.g. an authorization failure when pulling images from private repositories.

We must ensure that Docker registry credentials are correctly configured in Kubernetes secrets.

Network configuration should also be reviewed. Ensure that the Kubernetes nodes have network access to the Docker registry. Network policies or firewall rules might block access.

There are quite a few additional pitfalls such as problems with image tags. Ensure you are using the correct image tags. Latest tags might not always point to the expected image version.

If you're using a private registry you might be experiencing access issues. Make sure your credentials are up-to-date and the registry is accessible from all nodes in all regions.

Handling Node Issues

Node-related errors often point to physical or virtual machine issues. These issues can disrupt the normal operation of the Kubernetes cluster and need prompt attention.

To check node status use the command:

kubectl get nodes

We can then identify problematic nodes in the resulting output.

It's a cliché but sometimes rebooting nodes is the best solution to some problems. We can reboot the affected machine or VM. Kubernetes should attempt to "self-heal" and recover within a few minutes.

To investigate node conditions we can use the command:

kubectl describe node <node-name>

We should look for conditions such as MemoryPressure, DiskPressure, or NetworkUnavailable. These conditions provide clues about the underlying issue we should address in the node.

Preventive Measures

Node monitoring should be used to with tools such as Prometheus, Grafana to keep an eye on node health and performance. These work great for the low level Kubernetes related issues, we can also use them for high level application issues.

There are some automated healing tools such as the Kubernetes Cluster Autoscaler that we can leverage to automatically manage the number of nodes in your cluster based on workload demands. Personally, I'm not a huge fan as I'm afraid of a cascading failure that would trigger additional resource consumption.

Managing Missing Configuration Keys or Secrets

Missing configuration keys or secrets are common issues that disrupt Kubernetes deployments. Proper management of these elements is crucial for smooth operation.

We need to use ConfigMaps and secrets. These let us store configuration values and sensitive information securely. To avoid that we need to ensure that ConfigMaps and Secrets are correctly referenced in your pod specifications.

Inspect pod descriptions using the command:

kubectl describe pod <pod-name>

Review the output and look for missing configuration details. Rectify any misconfigurations.

ConfigMap and secret creation can be verified using the command:

kubectl get configmaps

and:

kubectl get secrets

Ensure that the required ConfigMaps and Secrets exist in the namespace and contain the expected data.

It's best to keep non-sensitive parts of ConfigMaps in version control while excluding Secrets for security. Furthermore, you should use different ConfigMaps and Secrets for different environments (development, staging, production) to avoid configuration leaks.

Utilizing Buildg for Interactive Debugging

Buildg is a relatively new tool that enhances the debugging process for Docker configurations by allowing interactive debugging.

It provides Interactive Debugging for configuration issues in a way that's similar to a standard debugging. It lets us step through the Dockerfile stages and set breakpoints. Buildg is compatible with VSCode and other IDEs via the Debug Adapter Protocol (DAP).

Buildg lets us inspect container state at each stage of the build process to identify issues early.

To install buildg follow the instructions on the Buildg GitHub page.

Conclusion

Debugging Kubernetes can be challenging, but with the right knowledge and tools, developers can effectively identify and resolve common issues. By understanding configuration problems, image pull errors, node issues, and the importance of ConfigMaps and Secrets, developers can contribute to more robust and reliable Kubernetes deployments. Tools like Buildg offer promising advancements in interactive debugging, further bridging the gap between development and operations.

As Kubernetes continues to evolve, staying informed about new tools and best practices will be essential for successful application management and deployment. By proactively addressing these common issues, developers can ensure smoother, more efficient Kubernetes operations, ultimately leading to more resilient and scalable applications.

Why is Kubernetes Debugging so Problematic?

Shai Almog — Tue, 04 Jun 2024 15:15:04 +0000

The Immutable Nature of Containers
The Limitations of kubectl exec
Avoiding Direct Modifications
Enter Ephemeral Containers
- Using kubectl debug
Practical Application of Ephemeral Containers
Security Considerations
Interlude: The Role of Observability
Command Line Debugging
Connecting a Standard IDE for Remote Debugging
Conclusion

Debugging application issues in a Kubernetes cluster can often feel like navigating a labyrinth. Containers are ephemeral by design, intended to be immutable once deployed. This presents a unique challenge when something goes wrong and we need to dig into the issue. Before diving into the debugging tools and techniques, it's essential to grasp the core problem: why modifying container instances directly is a bad idea. This blog post will walk you through the intricacies of Kubernetes debugging, offering insights and practical tips to effectively troubleshoot your Kubernetes environment.

The Immutable Nature of Containers

One of the fundamental principles of Kubernetes is the immutability of container instances. This means that once a container is running, it shouldn't be altered. Modifying containers on the fly can lead to inconsistencies and unpredictable behavior, especially as Kubernetes orchestrates the lifecycle of these containers, replacing them as needed. Imagine trying to diagnose an issue only to realize that the container you’re investigating has been modified, making it difficult to reproduce the problem consistently.

The idea behind this immutability is to ensure that every instance of a container is identical to any other instance. This consistency is crucial for achieving reliable, scalable applications. If you start modifying containers, you undermine this consistency, leading to a situation where one container behaves differently from another, even though they are supposed to be identical.

The Limitations of `kubectl exec`

We often start our journey in Kubernetes with commands such as:

$ kubectl -- exec -ti <pod-name>

This logs into a container and feels like accessing a traditional server with SSH. However, this approach has significant limitations. Containers often lack basic diagnostic tools—no vim, no traceroute, sometimes not even a shell. This can be a rude awakening for those accustomed to a full-featured Linux environment. Additionally, if a container crashes, kubectl exec becomes useless as there's no running instance to connect to. This tool is insufficient for thorough debugging, especially in production environments.

Consider the frustration of logging into a container only to find out that you can't even open a simple text editor to check configuration files. This lack of basic tools means that you are often left with very few options for diagnosing problems. Moreover, the minimalistic nature of many container images, designed to reduce their attack surface and footprint, exacerbates this issue.

Avoiding Direct Modifications

While it might be tempting to install missing tools on-the-fly using commands like apt-get install vim, this practice violates the principle of container immutability. In production, installing packages dynamically can introduce new dependencies, potentially causing application failures. The risks are high, and it's crucial to maintain the integrity of your deployment manifests, ensuring that all configurations are predefined and reproducible.

Imagine a scenario where a quick fix in production involves installing a missing package. This might solve the immediate problem but could lead to unforeseen consequences. Dependencies introduced by the new package might conflict with existing ones, leading to application instability. Moreover, this approach makes it challenging to reproduce the exact environment, which is vital for debugging and scaling your application.

Enter Ephemeral Containers

The solution to the aforementioned problems lies in ephemeral containers. Kubernetes allows the creation of these temporary containers within the same pod as the application container you need to debug. These ephemeral containers are isolated from the main application, ensuring that any modifications or tools installed do not impact the running application.

Ephemeral containers provide a way to bypass the limitations of kubectl exec without violating the principles of immutability and consistency. By launching a separate container within the same pod, you can inspect and diagnose the application container without altering its state. This approach preserves the integrity of the production environment while giving you the tools you need to debug effectively.

Using `kubectl debug`

The kubectl debug command is a powerful tool that simplifies the creation of ephemeral containers. Unlike kubectl exec, which logs into the existing container, kubectl debug creates a new container within the same namespace. This container can run a different OS, mount the application container’s filesystem, and provide all necessary debugging tools without altering the application’s state. This method ensures you can inspect and diagnose issues even if the original container is not operational.

For example, let’s consider a scenario where we’re debugging a container using an ephemeral Ubuntu container:

kubectl debug <myapp> -it <pod-name> --image=ubuntu --share-process --copy-to=<myapp-debug>

This command launches a new Ubuntu-based container within the same pod, providing a full-fledged environment to diagnose the application container. Even if the original container lacks a shell or crashes, the ephemeral container remains operational, allowing you to perform necessary checks and install tools as needed. It relies on the fact that we can have multiple containers in the same pod, that way we can inspect the filesystem of the debugged container without physically entering that container.

Practical Application of Ephemeral Containers

To illustrate, let’s delve deeper into how ephemeral containers can be used in real-world scenarios. Suppose you have a container that consistently crashes due to a mysterious issue. By deploying an ephemeral container with a comprehensive set of debugging tools, you can monitor the logs, inspect the filesystem, and trace processes without worrying about the constraints of the original container environment.

For instance, you might encounter a situation where an application container crashes due to an unhandled exception. By using kubectl debug, you can create an ephemeral container that shares the same network namespace as the original container. This allows you to capture network traffic and analyze it to understand if there are any issues related to connectivity or data corruption.

Security Considerations

While ephemeral containers reduce the risk of impacting the production environment, they still pose security risks. It’s critical to restrict access to debugging tools and ensure that only authorized personnel can deploy ephemeral containers. Treat access to these systems with the same caution as handing over the keys to your infrastructure.

Ephemeral containers, by their nature, can access sensitive information within the pod. Therefore, it is essential to enforce strict access controls and audit logs to track who is deploying these containers and what actions are being taken. This ensures that the debugging process does not introduce new vulnerabilities or expose sensitive data.

Interlude: The Role of Observability

While tools like kubectl exec and kubectl debug are invaluable for troubleshooting, they are not replacements for comprehensive observability solutions. Observability allows you to monitor, trace, and log the behavior of your applications in real-time, providing deeper insights into issues without the need for intrusive debugging sessions.

These tools aren't meant for everyday debugging, that role should be occupied by various observability tools. I will discuss observability in more detail in an upcoming post.

Command Line Debugging

While tools like kubectl exec and kubectl debug are invaluable, there are times when you need to dive deep into the application code itself. This is where we can use command line debuggers. Command line debuggers allow you to inspect the state of your application at a very granular level, stepping through code, setting breakpoints, and examining variable states. Personally, I don't use them much

For instance, Java developers can use jdb, the Java Debugger, which is analogous to gdb for C/C++ programs. Here’s a basic rundown of how you might use jdb in a Kubernetes environment:

Set Up Debugging: First, you need to start your Java application with debugging enabled. This typically involves adding a debug flag to your Java command. However, as discussed in my post here, there's an even more powerful way that doesn't require a restart:

java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -jar myapp.jar

Port Forwarding: Since the debugger needs to connect to the application, you’ll set up port forwarding to expose the debug port of your pod to your local machine. This is important as JDWP is dangerous:

kubectl port-forward <pod-name> 5005:5005

Connecting the Debugger: With port forwarding in place, you can now connect jdb to the remote application:

jdb -attach localhost:5005

From here, you can use jdb commands to set breakpoints, step through code, and inspect variables. This process allows you to debug issues within the code itself, which can be invaluable for diagnosing complex problems that aren’t immediately apparent through logs or superficial inspection.

Connecting a Standard IDE for Remote Debugging

I prefer IDE debugging by far. I never used JDB for anything other than a demo. Modern IDEs support remote debugging, and by leveraging Kubernetes port forwarding, you can connect your IDE directly to a running application inside a pod.

To set up remote debugging we start with the same steps as the command line debugging. Configuring the application and setting up the port forwarding.

Configure the IDE: In your IDE (e.g., IntelliJ IDEA, Eclipse), set up a remote debugging configuration. Specify the host as localhost and the port as 5005.
Start Debugging: Launch the remote debugging session in your IDE. You can now set breakpoints, step through code, and inspect variables directly within the IDE, just as if you were debugging a local application.

I show how to do it in IntelliJ/IDEA here.

Conclusion

Debugging Kubernetes environments requires a blend of traditional techniques and modern tools designed for container orchestration. Understanding the limitations of kubectl exec and the benefits of ephemeral containers can significantly enhance your troubleshooting process. However, the ultimate goal should be to build robust observability into your applications, reducing the need for ad-hoc debugging and enabling proactive issue detection and resolution.

By following these guidelines and leveraging the right tools, you can navigate the complexities of Kubernetes debugging with confidence and precision. In the next installment of this series, we’ll delve into common configuration issues in Kubernetes and how to address them effectively.

DEV Community: Shai Almog

Metal Default, A New Build Cloud, And A New Format

Metal is the default on iOS

The new Build Cloud console is now the default link

Upcoming attractions

Wrapping up

NFC, Crypto, Biometrics, And A New Build Cloud

A new Build Cloud UI — preview

Device APIs become first-class

Biometrics — PR #4987

Cryptography — PR #4994

NFC — PR #4996

cn1libs can now own simulator menus — and that changes Bluetooth

Bluetooth that you can actually debug

In-app purchase consistency — PR #4990

UTF-8: JDK-compatible replace semantics + a NEON ASCII fast path — PR #4989

Two long-standing JVM fixes

PR #4980 — Iterative GC mark to fix iOS stack overflow on deep graphs

PR #4985 — Don't rely on C arg eval order in PUTFIELD / MULTIANEWARRAY

Hardware keyboard and mouse on iOS and Android — PR #4982

Expanded CSS gradients and blurs — PR #4957

On Metal: the community got there first

Wrapping up

Skills, Java 17, And Theme Accents

Java 17 by default

AGENTS.md and the Codename One skill

Native theme accents

Metal follow-ups

Per-axis scale decomposition (#4939, fixes #3302)

Clip-under-rotation diagnostic (#4924, towards #3921)

iOS Metal colour space hint (#4909, fixes #4908)

The new translateMatrix API

String API: replace(CharSequence, CharSequence), replaceAll, replaceFirst

iOS push permission no longer fires at app launch

Skin Designer FAQ follow-up

Wrapping up

Metal and Skins

How we think about quality

Sticky headers were half baked, and that was by design

The SIMD bug, which was my mistake

Metal is here

The end of the skin downloader

How the wizard works

Eating our own dog food

iOS multi-line TextArea: Return as Done

Diagnostics for status-bar tap scroll-to-top

Simulator: Dark / Light mode toggle

Heads-up: weekend backend maintenance

Warning: Android 16 will effectively disallow locking orientation

Why the version jumped to 7.0.242

Wrapping up

Liquid Glass, Material 3, And A Lot Of Plumbing

Try it right now in the Playground

The new native themes

iOS Modern

Android Material 3

Translucent surfaces

Runtime palette overrides

In the simulator

On devices

Sticky headers

How we test this

Bugs and misc features from this week

Where this is going — and a thank-you

Front End Debugging Part 3: Networking

Network Debugging Powerhouse

Re-Issuing and Modifying Requests

cURL and Postman

Throttling and Debugging Race Conditions

Why Throttling Matters:

How to Use

Managing State with Storage Tools

Challenges of State Management

Analyzing Request and Response Headers

Debugging in Incognito Mode: Limitations and Best Practices

Connecting the Front-End to the Database

Final Word

Front End Debugging Part 2: Console.log() to the Max

Understanding Front-End Logging vs. Back-End Logging

Leveraging Console Log Levels

PR #4985 — Don't rely on C arg eval order in `PUTFIELD` / `MULTIANEWARRAY`

The new `translateMatrix` API

String API: `replace(CharSequence, CharSequence)`, `replaceAll`, `replaceFirst`

Instant Debugging with the `debugger` Keyword

The Limitations of `kubectl exec`

Using `kubectl debug`