DEV Community: M S

UnicodeFix: The Day Invisible Characters Broke Everything

M S — Fri, 25 Apr 2025 04:00:00 +0000

From Frustration to Fix

This all started maybe five days ago with a Python script that just wouldn’t run.

Not because of logic errors. Not bad syntax.

No, this was something worse — something invisible.

I had copied some code straight out of Cursor’s chat panel into a file. Looked fine. Ran it.

Boom: IndentationError: unexpected indent.

I stared at it for a while, pulled my hair a bit. I knew the indentation was fine.

It was flat code — no nesting, no tabs. I even did a global find-and-replace on the leading spaces.

Same result.

The idea that it could be Unicode? Didn’t even occur to me.

Eventually, Cursor’s own assistant piped up and said:

“Oh, you’ve got hidden Unicode spaces at the beginning of each line. Strip those out and it’ll run.”

And sure enough — it did.

That one invisible character wasted hours.

It looked like whitespace. It acted like whitespace. But it broke the interpreter.

And that’s when I started looking around.

Turns out, I wasn’t the only one.

I hit Reddit and started seeing post after post:

“Cursor broke my code with invisible stuff.”

“VS Code is inserting Unicode garbage and my linter can’t catch it.”

“Why does AI code have these weird characters no human would ever type?”

One comment hit a nerve:

“AI code has a Unicode accent now. Professors can tell.”

And that’s when it clicked. This isn’t just annoying. It’s serious.

These invisible Unicode characters — zero-width spaces, left and right smart quotes, BOMs, and friends — can break everything from Python scripts to YAML configs to HTML layouts.

In some cases, they even flag work as AI-generated.

Imagine working hard on something, refining it, shaping it — only for it to be dismissed because the quotes aren’t “ASCII enough.”

It brought back flashbacks from Japan, too — the dreaded mojibake , when UTF-8 and UTF-16 would collide and your characters would turn into incomprehensible glyph soup. Rendering issues. Encoding mismatches. Terminal output that looked like alien language.

These weren’t just technical nuisances. They were trust-breakers.

I realized: we needed a fix. A proper tool. Something immediate , local , and clean — to give people back control over their text.

Dictation = Velocity

One thing made this project move faster than anything else: I wasn’t typing.

I’d just wrapped up my Willow Voice trial on macOS and was on the fence. I figured I’d give it one more go.

And it was like flipping a switch.

Suddenly, instead of chasing ideas with my keyboard, I could speak them aloud as fast as they came to me.

Function names. Log messages. TODOs. Architecture decisions. Even this blog post — much of it dictated, not typed.

Willow handled the flow like a pro: no filler, no “uhhh”, no lag between thought and code.

In a way, it made collaboration with my AI assistant more fluid too — like having a real-time partner in a shared headspace. I wasn’t juggling syntax; I was thinking about what I wanted to build. The layers, the UX flow, the traps to avoid — and the fixes.

Honestly? That single shift cut development time by at least 30%. Maybe more.

Getting It to Click

Once the core functionality was solid — stripping quotes, normalizing dashes, vaporizing zero-width ghosts — the question was: how do we make this easy for anyone to use?

I didn’t want this to live only in the CLI. Sure, it could work in pipelines, inside Vim via %!, or piped from pbpaste. But not everyone lives in a terminal.

That’s when I turned to macOS Shortcuts — and that’s where things got weird.

Shortcuts seemed simple at first. Just a shell command, right?

But it quickly became clear: to pass file arguments into a proper Python environment with a conda-managed virtualenv, I needed a full login shell, .bashrc execution, and some clever quoting tricks.

One misstep and the shortcut would fail silently, or worse — run without activating the environment.

Even getting arguments passed in cleanly required tweaking the bash -l -c incantation to include both the Python script and the parameters outside the quoted string. Subtle stuff, but critical.

And yet, when it finally worked — right-clicking one or more files in Finder, selecting “Strip Unicode”, and instantly generating .clean.txt outputs — it felt like we’d stitched together something native. Something that belonged there all along.

It was clean, it was fast, and it worked exactly as you’d expect.

Mission complete? Not quite — but the bones were solid.

The Problem Was Bigger Than Expected

At first, it seemed simple enough: strip out invisible Unicode junk and normalize spacing.

But like most rabbit holes in computing, the deeper we looked, the more twisted it got.

It wasn’t just zero-width spaces or sneaky joiners that were breaking things.

It was smart quotes , em-dashes , weird hyphens , and other typographically “fancy” Unicode that had no business being inside a code block. Stuff that wouldn’t break your editor, but would break your parser.

Quotes that looked like this — “ ” — instead of this — “ “.

Characters that rendered fine on macOS or a web page, but blew up inside Python, YAML, or Markdown.

And the real kicker? They were almost impossible to spot unless you were really looking — or digging with a hex editor.

I realized it wasn’t just an inconvenience. It could affect everything:

Code execution
Web rendering
AI detection
Text indexing
Compatibility between systems

You name it.

Redditors were screaming about it.

I was quietly screaming too — holding open a hex editor and shaking my head.

The more we dug, the deeper it got.

I had my assistant generate a whole suite of test files: filled to the brim with Unicode landmines.

Some were invisible. Some were misleading. All of them were traps — not academic samples, but real-world mimics of AI-generated content copied from ChatGPT windows, web pages, and embedded chat panels.

We weren’t just “cleaning up whitespace” anymore.

We were building a normalization engine — one that could take in all that appearance-driven noise and distill it down into clean, predictable, ASCII-safe content without breaking intent.

A left or right smart quote? Normalized into ".
An en-dash or em-dash? Flattened into a -.

No semantic loss. Just clarity, restored.

This wasn’t a one-off patch anymore — it was shaping up to be a reliable bridge between messy, over-encoded text and what a compiler, parser, or linter actually expects.

That’s when we knew: we weren’t just solving our problem.

We were solving a growing, urgent problem for everyone — and doing it fast. —

Building a Smarter, Tougher Tool

The first phase was to make a basic pipe: read in a file or a stream, scrub out the bad Unicode, normalize smart punctuation, and write a clean file.

At one point, we considered doing it with pure sed, awk, or some bash wizardry. But once you start needing context-aware replacements — like turning a left quote and a right quote into a single plain ASCII quote — those tools hit a wall fast.

Python was the right hammer for this nail.

We kept the script lightweight.

No crazy dependencies. Just the standard argparse, re, a tiny bit of filesystem handling, and the excellent unidecode library to fill in the heavy lifting.

After a few iterations — and a lot of piping weird Unicode test blobs through it — the core script was done. It:

Normalized all smart quotes and dashes
Stripped hidden Unicode like zero-width spaces and BOMs
Worked on any file or STDIN
Wrote clean .clean.txt versions of files

Simple. Fast. Solid.

Shortcutting Into the Future

Now, getting the command-line tool working was great —

but how do you make it truly frictionless for users?

Enter macOS Shortcuts.

This was an unexpected turn. I hadn’t seriously touched Shortcuts before — not since the old Automator days.

But digging into it, I realized Shortcuts could:

Trigger from the Finder right-click menu
Pass selected files as arguments
Run shell scripts in the background without opening a terminal window

It sounded perfect… and it almost was.

Shortcuts is powerful, but getting it to correctly invoke a bash login shell, preserve the environment, and pass file arguments properly took some serious trial and error.

We had to:

Force Shortcuts to launch bash -l -c
Source the .bashrc to set up the Conda Python environment
Handle quoting and argument expansion carefully so multiple files could be passed cleanly

There’s no real documentation on this.

You either figure it out or you don’t.

But when it finally worked — being able to select a bunch of files in Finder, right-click, and instantly strip all Unicode quirks without even touching the Terminal — that was magic.

It felt like something that belonged in macOS all along.

Remote Debugger With VSCode and Its Derivatives

M S — Tue, 08 Apr 2025 22:30:00 +0000

How to Debug Remotely with VSCode

Recently, someone asked how to connect to a remote process and debug it. If you’re using VSCode—or any of its forks such as Cursor or Windsurf—the following method can save you hours of troubleshooting compared to relying on log files or print statements. A proper debugger lets you inspect variables, step through code, and quickly identify issues. Here’s how I set up remote debugging.

The Setup

Setting up remote debugging with VSCode is surprisingly simple and can often be done in just a few minutes. I encapsulated the necessary code into a small Python module that you can import into your projects.

Launch Configuration

First, configure your launch.json in VSCode with the following configuration:

    {
        "name": "Python: Attach to Process",
        "type": "debugpy",
        "request": "attach",
        "connect": {
            "host": "localhost",
            "port": 5678
        },
        "pathMappings": [
            {
                "localRoot": "${workspaceFolder}",
                "remoteRoot": "."
            }
        ]
    }

This tells VSCode to attach to a debug server running on port 5678. The pathMappings field ensures that file paths on your remote system map correctly to your local workspace.

The Debug Module

Next, create a module (for example, debug_my.py) with the following code:

    import debugpy
    from modules.logging_colors import logger

    def remote_debugger_activate():
        """
        Starts the debugpy server. In a real application, ensure this is secured properly.
        """
        # Listen on all interfaces (0.0.0.0); in production, restrict this if possible.
        debugpy.listen(('0.0.0.0', 5678))
        logger.critical("Debugging enabled. Waiting for debugger to attach...")
        debugpy.wait_for_client() # The process will block here until a debugger attaches.
    logger.critical("Debugger attached.")

Setup your code to start the debugger

Then, in your main Python application, import and call the function where you need to start remote debugging:

from debug_my import remote_debugger_activate

# ... your code ...

if args.remote_debug:
    remote_debugger_activate()

# ... rest of your code ...

Running the Debugger

Start your Python application on the remote host (or locally under a different user).
In VSCode, go to the “Run and Debug” view and select your “Python: Attach to Process” configuration.
Press the run button in VSCode.
The remote process will block at debugpy.wait_for_client() until VSCode attaches.
Once attached, your code continues execution, and you can begin interactive debugging.

Security Considerations

Keep in mind that exposing a debug server can be a significant security risk:

Restrict Network Access: In a secure environment, bind the debug server only to localhost or use SSH tunneling.
Disable Debugging When Not Needed: Always disable remote debugging when it’s not actively required for troubleshooting.
Use a Non-Default Port: If necessary, choose a port that is not commonly targeted.

Troubleshooting Tips

Port Conflicts: Verify that port 5678 is available or change it if necessary.
Connectivity Issues: Ensure your remote machine allows incoming connections on the chosen port.
Log Verbosity: Increase your debugger or application log level for more detailed diagnostics if debugging fails to attach.

Why Remote Debugging?

Remote debugging provides an interactive, real-time view into your application’s state. Rather than sifting through log files or inserting temporary print statements, a debugger lets you:

Step through your code.
Inspect variables.
Evaluate expressions on the fly.

This capability can significantly speed up bug resolution and reduce downtime.

Further Resources

I hope this guide helps you set up a more efficient and effective remote debugging workflow using VSCode or its variants. Happy debugging!

Additional Information

Security Considerations

Remote debugging can be a security risk if left enabled in production. You might add a short section on securing the debug server, for example:

Restrict Network Access: Mention that in a production setting you should bind the debug server to a specific interface or use firewall rules so that only trusted hosts can connect.
Use Authentication (if available): Debugpy and similar tools may provide options for authentication or you can tunnel the debug connection over SSH. Explain briefly how to do that, for example, by creating an SSH tunnel:

    ssh -L 5678:localhost:5678 user@remote_host

Disable Debugging When Not Needed: Emphasize that remote debugging should only be enabled when needed for troubleshooting and then promptly disabled.

More Detailed Explanation of pathMappings

Many users might not be familiar with pathMappings in the launch.json. You could explain how localRoot and remoteRoot work—i.e. that they help map the file system structure between the remote host and your local development environment, ensuring breakpoints are placed in the correct files.

Troubleshooting Tips

Add a brief section that covers common pitfalls:

Port Conflicts: If the specified port (5678 in your example) is in use by another process, mention that users should either free that port or choose a different one.
Connection Timeouts: If the debugger fails to attach, advise readers to verify that the remote machine can reach the debug server (e.g. by checking connectivity or firewall settings).
Log Verbosity: Explain that increasing the debug level (using your -d option) might provide helpful logs if something goes wrong.

Benefits of Remote Debugging

You might add a paragraph or two about why remote debugging is so valuable:

Reduced Downtime: Instead of relying on print statements, remote debugging allows interactive inspection of variables and flow control, saving significant time.
Enhanced Productivity: Developers can step through code in real time, even on remote servers, which can be crucial when dealing with production issues or complex distributed systems.

Submitted (Again) to the Apple App Store

M S — Thu, 20 Mar 2025 04:00:00 +0000

UPDATE: LogGPT Available Now on the Apple App Store Now!

I’ve finally got LogGPT approved by Apple and it is now available on the App Store.

The Long Road to Resubmission

After what feels like an eternity, I’ve finally resubmitted the Safari extension to the Apple App Store. If you’ve ever dealt with Apple’s review process, you know the frustration that comes with seemingly arbitrary rejections. And yes, that happened to me.

Initially, my app was rejected because the screenshot looked too much like ChatGPT. Well… yeah, because it’s a Safari extension for ChatGPT! The extension simply lets you download your ChatGPT conversation history as a JSON file. Nothing more, nothing less.

Fixing the Issues & Jumping Through Hoops

Getting this app back into Apple’s review queue was not straightforward. Here’s a quick summary of the madness:

Name Change : The original name was identical to an existing open-source MIT-licensed project that provides a similar extension for Firefox and Chrome. Even though there was no Safari version, I decided to change the name to avoid confusion and conflicts.
The Icon Issue : Apple rejected my submission because the icon looked too much like ChatGPT’s branding and was used by other apps. I had to create a new icon to meet their guidelines.
Privacy Policy & Website Updates : Since I changed the app’s name, I also had to update my privacy policy and make website changes to reflect the new branding. More work I wasn’t expecting.
New Apple Submission Hurdles : Apple, true to its Steve Jobs-era philosophy of always having “one more thing” , kept introducing additional steps I had to complete. Just when I thought I was done, they added another requirement.
New Info.plist Property for Encryption : Apple now requires a new data encryption property in Info.plist. I had to manually add it to comply with their updated security policies.
Chinese App Store Removal : Since ChatGPT is not allowed in China , I had to remove the app from the Chinese App Store entirely. While users there could still purchase it, it wouldn’t work anyway due to ChatGPT restrictions.
Regions Affected : I assume that Hong Kong and Shanghai are also included in this restriction since they are part of China, though I wouldn’t be surprised if an alternative ChatGPT-like service exists that the extension could work with.
Final Submission & Price : After fixing these issues, I resubmitted the app with a price of $1.99 —mostly to help offset the annual Apple Developer Tax ($99/year).

What This Extension Does

To reiterate, this is a simple, no-nonsense Safari extension that lets you export your ChatGPT conversations to JSON. No ads, no tracking—just a clean way to download your chat logs for archival or content reuse. The file is saved in your Downloads folder using the conversation’s title from OpenAI.

I personally use this extension for:

Printing chat logs for reference.
Extracting useful responses and formatting them into Markdown.
Archiving conversations for future use.
Splitting exported files and uploading them into a new ChatGPT session to continue conversations seamlessly (I’ll be writing more about that technique soon).

Used Just Today for AI Context Transfer

An unexpected but incredibly useful scenario happened just this morning: we hit the maximum conversation length in ChatGPT. Instead of losing our discussion, we used this extension to export the conversation to JSON , split it into Markdown, and reload it into a fresh session— seamlessly continuing without missing a beat. This proves how useful it is for those working with AI over long discussions, ensuring no context is lost.

Extra Tools for Command Line Users

If you’re comfortable with the command line, check out my VenvUtil project. It includes tools to:

Convert exported JSON files into HTML or Markdown.
Format and clean up conversations for printing.
Make archiving and retrieval easier.

Feeling Adventurous?

If you want to build it yourself, you may download the source code as well as a pre-compiled binary of the app on my GitHub Repository page. There is a release there as well you may download and install it on your system.

Support My Work

This extension is priced at $1.99 , but if you’d like to support me further, you can do so here:

This has been an exhausting process, and I’m just glad to finally have it back in Apple’s queue. Fingers crossed for a smooth approval this time!

If you are looking fro someone to help you with a project, I am available for hire. I have a wide range of skills and can be a great asset to your team. My specialties are Unix/Linux/macOS system and database administration, automation, and application development and am rather adept with Python, Bash, and a variety of other tools.

RE: Submitted (Again) to the Apple App Store

M S — Thu, 20 Mar 2025 04:00:00 +0000

Now available in the Apple AppStore - LogGPT

The Long Road to Resubmission

Fixing the Issues & Jumping Through Hoops

Getting this app back into Apple’s review queue was not straightforward. Here’s a quick summary of the madness:

Name Change : The original name was identical to an existing open-source MIT-licensed project that provides a similar extension for Firefox and Chrome. Even though there was no Safari version, I decided to change the name to avoid confusion and conflicts.
The Icon Issue : Apple rejected my submission because the icon looked too much like ChatGPT’s branding and was used by other apps. I had to create a new icon to meet their guidelines.
Privacy Policy & Website Updates : Since I changed the app’s name, I also had to update my privacy policy and make website changes to reflect the new branding. More work I wasn’t expecting.
New Apple Submission Hurdles : Apple, true to its Steve Jobs-era philosophy of always having “one more thing” , kept introducing additional steps I had to complete. Just when I thought I was done, they added another requirement.
New Info.plist Property for Encryption : Apple now requires a new data encryption property in Info.plist. I had to manually add it to comply with their updated security policies.
Chinese App Store Removal : Since ChatGPT is not allowed in China , I had to remove the app from the Chinese App Store entirely. While users there could still purchase it, it wouldn’t work anyway due to ChatGPT restrictions.
Regions Affected : I assume that Hong Kong and Shanghai are also included in this restriction since they are part of China, though I wouldn’t be surprised if an alternative ChatGPT-like service exists that the extension could work with.
Final Submission & Price : After fixing these issues, I resubmitted the app with a price of $1.99 —mostly to help offset the annual Apple Developer Tax ($99/year).

What This Extension Does

I personally use this extension for:

Printing chat logs for reference.
Extracting useful responses and formatting them into Markdown.
Archiving conversations for future use.
Splitting exported files and uploading them into a new ChatGPT session to continue conversations seamlessly (I’ll be writing more about that technique soon).

Used Just Today for AI Context Transfer

Extra Tools for Command Line Users

If you’re comfortable with the command line, check out my VenvUtil project. It includes tools to:

Convert exported JSON files into HTML or Markdown.
Format and clean up conversations for printing.
Make archiving and retrieval easier.

Feeling Adventurous?

Support My Work

This extension is priced at $1.99 , but if you’d like to support me further, you can do so here:

This has been an exhausting process, and I’m just glad to finally have it back in Apple’s queue. Fingers crossed for a smooth approval this time!

Announcing TorchDevice 0.0.5 Beta – Transparent Hardware Redirection for PyTorch

M S — Wed, 05 Mar 2025 05:00:00 +0000

New Version soon to be released

This is the older version in main, bug there will soon be a refactored and well tested version of this soon.

We’re pleased to announce the release of TorchDevice 0.0.5 Beta , a significant milestone in simplifying hardware compatibility for PyTorch applications. This release introduces robust enhancements, thorough testing, and a powerful new CPU override feature to ensure seamless integration across CUDA and Apple Silicon (Metal) hardware.

Why TorchDevice?

PyTorch is extensively optimized for CUDA hardware. However, developers using Apple Silicon devices face compatibility challenges due to PyTorch’s CUDA-centric assumptions. TorchDevice bridges this gap by transparently redirecting PyTorch device calls to the correct backend (CUDA, Metal, or CPU) without needing any manual intervention or code modifications.

What’s New in 0.0.5?

Apart from improving testing, more functionality is now working in this version. Compatibility with Streams, Events, and Context management.

CPU Override Feature

The highlight of this release is the new CPU override functionality , enabling developers to explicitly force CPU usage, invaluable for:

Debugging and testing GPU-dependent code on a CPU.
Ensuring consistent behavior across various testing environments.
Resource management by reserving GPU resources for specific tasks.

Activation is straightforward:

import torch
import TorchDevice

# Forces CPU usage universally
device = torch.device('cpu:-1')

# Operations below explicitly use CPU despite CUDA or MPS availability
tensor = torch.randn(5, 5) # CPU-based tensor
model = torch.nn.Linear(10, 5).to('cuda') # Redirected to CPU automatically

Expanded and Improved Testing

TorchDevice 0.0.5 is extensively tested using HuggingFace Transformers, successfully passing most test scenarios and ensuring robust compatibility with widely-used models. Further testing and refinements continue, enhancing stability and coverage.

Comprehensive Documentation & Examples

We’ve expanded the documentation to clearly guide developers through installation, configuration, and troubleshooting, making adoption effortless.

Detailed README
Practical usage examples and tutorials
Open community support and contributions via GitHub issues and pull requests

What’s Next?

Our future plans include:

Enhanced runtime flexibility for CPU overrides
Granular control over device selection per operation
Performance optimizations for mixed-device workflows

Community and Support

TorchDevice is actively developed and maintained, and your support helps us accelerate improvements and features. If you or your organization finds value in TorchDevice, consider supporting development through sponsorship or donations.

Help Support Our Projects

Checkout my other projects at my site
Distributed Thinking Systems

If you find this useful help support my work;

Your feedback drives our roadmap. Let’s make PyTorch more accessible and efficient for everyone.