{"id":32,"date":"2026-09-05T09:55:25","date_gmt":"2026-09-05T09:55:25","guid":{"rendered":"https:\/\/www.0xprathamesh.com\/blog\/?p=32"},"modified":"2026-09-21T06:05:00","modified_gmt":"2026-09-21T06:05:00","slug":"ascii-smuggling-unicode-nfkc","status":"publish","type":"post","link":"https:\/\/www.0xprathamesh.com\/blog\/ascii-smuggling-unicode-nfkc\/","title":{"rendered":"ASCII Smuggling: The Invisible Text NFKC Leaves Behind"},"content":{"rendered":"

ASCII smuggling hides characters inside text that can look ordinary on screen. Calling Unicode\u2019s NFKC normalisation function does not automatically remove them.<\/strong> In the local Python example below, an invisible tag character survives normalisation and still breaks a literal word match. That distinction matters anywhere email, retrieved documents or support tickets feed an AI assistant.<\/p>\n

Consider a security review of an assistant that reads incoming requests. The reviewer sees a familiar word. The application receives that word with an extra character in the middle. Both components are working with the same message, but they have different evidence about its contents. A screenshot cannot show the whole problem.<\/p>\n

I would start the review with that small disagreement. Before debating how well the model resists a hidden instruction, establish what text actually reaches it, what your filter checked, and what the human approval screen displays.<\/p>\n

What Microsoft reported, and when<\/h2>\n

On 3 September 2026, Microsoft Security Research, Noam Kochavi and Sarah Wolstencroft reported<\/a> a phishing campaign that inserted invisible Unicode tag characters into financial keywords. Research into hidden instructions for AI assistants had surfaced a different use: disrupting text analysis of email lures.<\/p>\n

The report describes a telemetry increase beginning on 9 February 2026 and a sharp reduction in the technique\u2019s high-volume phase after 15 May. September is the disclosure date, not the start of the observed activity. Microsoft says more than 99% of the messages were flagged by other protection layers. Signature-hit volume should therefore not be read as successful delivery, victim count or proof that an entire mail-security product was bypassed.<\/p>\n

The engineering question raised by the report is worth testing independently: when someone says their pipeline \u201cnormalises Unicode\u201d, which transformation do they mean?<\/p>\n

What ASCII smuggling means<\/h2>\n

In this context, ASCII smuggling uses non-rendering Unicode tag characters to carry or interrupt text while leaving little visible evidence. The name can be confusing: these are Unicode code points outside the ordinary ASCII range. Displaying a character and retaining it in a string are separate operations.<\/p>\n

A code point<\/em> is a numbered entry in Unicode. A rendered symbol may consist of several code points, and a code point may produce no visible mark. This is why a visual check, a string-length check and a text-matching operation can disagree without any of them crashing.<\/p>\n

\n\"One
Original illustration of the local example. The highlighted U+E0020 label is deliberately visible; the image contains no hidden instruction.<\/figcaption><\/figure>\n

A small experiment: NFKC preserves the tag<\/h2>\n

This example uses one harmless word and one explicit escape sequence. It sends nothing, contacts no service and makes no claim about a particular model or commercial filter. The local run used Python 3.14.0 with Unicode database version 16.0.0.<\/p>\n

import unicodedata\n\nsample = \"fun\\U000E0020ding\"\nnormalised = unicodedata.normalize(\"NFKC\", sample)\n\nprint(\"Unicode database:\", unicodedata.unidata_version)\nprint(\"Code points:\", len(sample))\nprint(\"Literal match:\", \"funding\" in sample)\nprint(\"NFKC unchanged:\", normalised == sample)\nprint(\"NFKC match:\", \"funding\" in normalised)\n\nfor position, char in enumerate(sample):\n    if 0xE0000 <= ord(char) <= 0xE007F:\n        print(position, f\"U+{ord(char):05X}\",\n              unicodedata.name(char, \"UNASSIGNED\"))<\/code><\/pre>\n
The observed output was:<\/p>\n
Unicode database: 16.0.0\nCode points: 8\nLiteral match: False\nNFKC unchanged: True\nNFKC match: False\n3 U+E0020 TAG SPACE<\/code><\/pre>\n
The seven visible letters are separated by an eighth code point. Python\u2019s substring operation requires the characters of the search term to be contiguous. The inserted character interrupts that sequence, and NFKC leaves this sample unchanged. The output establishes that narrow behaviour in the recorded runtime; it does not establish how a tokenizer, a browser font or a security classifier will behave.<\/p>\n
Unicode Standard Annex #15<\/a> defines normalisation around canonical and compatibility equivalence. NFKC performs compatibility decomposition followed by canonical composition. It can make certain alternative representations comparable, but it is not a general instruction to delete every non-rendering character. Unicode also cautions against applying compatibility normalisation blindly to arbitrary text because meaningful distinctions can be lost.<\/p>\n
Python\u2019s unicodedata<\/code> documentation<\/a> makes the operation and database version inspectable. Recording that version alongside the fixture is useful when the ingestion service later upgrades its runtime. A comment saying \u201csanitised\u201d gives the next engineer much less to verify.<\/p>\n
Expose the difference before changing the text<\/h2>\n
For investigation, I would preserve the original and create a readable inspection view. The following function renders tag-block characters as visible labels. This is a diagnostic display, not an instruction decoder or a production sanitiser.<\/p>\n
def inspect_tags(text):\n    return \"\".join(\n        f\"[U+{ord(char):05X}]\"\n        if 0xE0000 <= ord(char) <= 0xE007F\n        else char\n        for char in text\n    )\n\nprint(inspect_tags(sample))\n# fun[U+E0020]ding<\/code><\/pre>\n
Showing the position matters. A record saying \u201cone unusual character\u201d gives a reviewer little context. Showing the extra code point inside the word explains the failed match immediately. In a real interface, escape HTML and handle other control characters as well; this short function deliberately covers only the range being examined.<\/p>\n
Now make the transformation explicit. For this isolated fixture, removing tag-block characters from a separate comparison copy restores the literal match:<\/p>\n
comparison = \"\".join(\n    char for char in sample\n    if not 0xE0000 <= ord(char) <= 0xE007F\n)\nprint(\"Comparison match:\", \"funding\" in comparison)\n# Comparison match: True<\/code><\/pre>\n
That result gives you a regression test for one text-processing condition. It says nothing about whether \u201cfunding\u201d is a malicious word. The point is to make a documented comparison rule behave consistently while retaining the original input for analysis.<\/p>\n
A flag emoji is a useful false-positive test<\/h2>\n
Deleting every tag character from every input is a poor general product policy. Unicode\u2019s emoji specification<\/a> defines legitimate tag sequences. England, Scotland and Wales flag emoji use a base flag followed by tag characters and a terminating tag. Removing the tags can change the displayed flag.<\/p>\n
This gives the review a useful pair of fixtures: the interrupted word and a valid flag sequence. Both contain tag characters. Only one is being used here to interfere with word matching. The surrounding sequence and the field\u2019s purpose determine how the application should handle it.<\/p>\n
A customer comment field, a payment beneficiary identifier and a forensic evidence view need different rules. An identifier may have a narrow permitted character set. A comment should preserve ordinary language and emoji. An evidence view needs to expose surprising content without silently destroying it. Make those choices per field and publish them for the people who maintain the parsers.<\/p>\n
For an AI application serving Indian users, an ASCII-only fallback is particularly costly. Names and messages may legitimately use Devanagari and other scripts. Include real language requirements in the test corpus. \u201cEnglish test messages still work\u201d is too small a compatibility check for a multilingual product.<\/p>\n
Make each consumer\u2019s input traceable<\/h2>\n
Imagine a support assistant that classifies a ticket and drafts a response. This is a representative design, not a customer incident. The application may extract text from HTML, run a classifier, retrieve related tickets, assemble a model request and display a preview. A different transformation at each step can leave the operator approving a view that never matches the model\u2019s input.<\/p>\n
I would keep a record of the extraction and transformation version used for each derived view. Preserve the original in storage with appropriate access and retention controls, and associate the derived text with that record. The system should be able to answer which view was classified, which view reached the model, and which view a reviewer saw.<\/p>\n
\n\"Proposed
A proposed design for tracing text transformations. Separate views are useful when their relationship to the source is recorded and the downstream consumer is known.<\/figcaption><\/figure>\n
Keep transformations bounded. Do not recursively decode arbitrary content until it resembles readable instructions. That can turn obscure material into a new instruction surface and consume resources on malformed input. Specify which encodings the application accepts, enforce size limits, and surface a parsing failure instead of quietly sending an unexamined fallback into the model.<\/p>\n
Preservation also has a cost. Raw tickets can contain personal or confidential information. A diagnostic feature should not copy every message into a broad telemetry channel. Reuse the controlled evidence store and give engineers synthetic fixtures when they only need to reproduce parser behaviour. The previous field note on AI log custody<\/a> covers the permissions and retention questions around that store.<\/p>\n
Readable instructions can still be untrusted<\/h2>\n
After a text-processing fix, the assistant may still encounter a perfectly visible request to change its task. A customer ticket can contain instructions addressed to the assistant even when every character is ordinary English. Text cleanup cannot decide whether that customer is allowed to change an internal workflow.<\/p>\n
OWASP\u2019s prompt-injection guidance<\/a> recommends limiting privileges, separating external content and requiring human involvement for high-risk actions. In the support example, the backend should independently verify account scope and permission before changing a record. A ticket\u2019s contents should not grant the assistant broader access.<\/p>\n
For approval screens, show the actual proposed operation and its affected records. An innocent-looking excerpt from the ticket is weak evidence for approving a consequential action. Bind the approval to the operation the backend will execute, so a later model turn cannot quietly substitute a different destination or account.<\/p>\n
A practical regression set<\/h2>\n
Start with cases small enough that an engineer can explain every outcome. Then run them through the complete application path, including the display and the model-request assembly. These are proposed acceptance checks, not measurements of any vendor\u2019s product.<\/p>\n
\n\n\n\n\n\n\n\n\n\n\n
Fixture<\/th>\nWhat to verify<\/th>\nUseful evidence<\/th>\n<\/tr>\n<\/thead>\n
Ordinary text<\/td>\nThe documented comparison still matches and the display stays readable.<\/td>\nInput, transformation version and decision.<\/td>\n<\/tr>\n
One tag inserted inside a harmless word<\/td>\nInspection exposes the position; the comparison policy has an explicit outcome.<\/td>\nOriginal and escaped views, plus the consumer’s actual input.<\/td>\n<\/tr>\n
A valid subdivision flag emoji<\/td>\nA legitimate sequence does not become a blanket reason to reject the message.<\/td>\nSequence classification and the resulting display.<\/td>\n<\/tr>\n
Names and text in supported scripts<\/td>\nProcessing preserves the product’s language requirements.<\/td>\nA representative multilingual corpus reviewed by language users.<\/td>\n<\/tr>\n
Visible instructions inside a customer ticket<\/td>\nExternal text cannot authorise a privileged operation.<\/td>\nThe backend permission decision and affected account.<\/td>\n<\/tr>\n
Parser failure or oversized content<\/td>\nThe failure path does not forward an unexamined substitute.<\/td>\nError state, bounded processing and a reviewable fallback.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
Repeat the set after changes to HTML extraction, Unicode libraries, retrieval processing or the model integration. Where model behaviour varies, report repeated-run outcomes separately from deterministic parser checks. A string comparison that passes every time and an assistant that usually follows its task are different measurements.<\/p>\n
Questions worth settling in the design review<\/h2>\n
Does NFKC remove invisible Unicode characters?<\/h3>\n
NFKC is not a general removal function. In the recorded Python test, U+E0020 TAG SPACE survived unchanged. Check the exact characters and runtime your application handles, then implement and test a separate policy where removal or visible inspection is required.<\/p>\n
Do invisible characters always mean prompt injection?<\/h3>\n
No. Legitimate emoji sequences use tag characters, and unusual text may appear in security research or diagnostic material. Evaluate the sequence, source and intended use. A character-range match is evidence to interpret.<\/p>\n
Will this fix stop prompt injection?<\/h3>\n
It can address a specific discrepancy in text handling. Prompt injection also works through readable content, so actions still need independent authorisation and restricted permissions. Measure the text-processing behaviour separately from the assistant’s response.<\/p>\n
What should a CTO ask for first?<\/h3>\n
Ask the team to trace one synthetic message through the filter, the model request and the approval screen. Have them expose hidden code points and name every transformation. Then check whether any difference can affect an action the assistant is allowed to take.<\/p>\n
The most useful result of that exercise may be a very ordinary bug: a parser running in the wrong order, an outdated comparison rule, or a preview built from a different field. Those are concrete fixes a team can ship and verify. Keep the fixtures beside the integration so the next update has to preserve the behaviour you agreed on.<\/p>\n
<\/p>\n
One next action<\/h2>\n
If you run an agent that executes tools, AgentCrash<\/a> is an open-source harness that crash-tests it in a synthetic workplace: synthetic tools, visual evidence and regression tests. It is built for exactly this problem, turning a containment claim into something you can run against the system instead of arguing about it in review.<\/p>\n","protected":false},"excerpt":{"rendered":"
ASCII smuggling can hide Unicode in plain-looking text. A tested Python example shows why NFKC is insufficient and what AI ingestion pipelines should check.<\/p>\n","protected":false},"author":0,"featured_media":33,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[12,47,50,49,48],"class_list":["post-32","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-blog-notes","tag-ai-security","tag-ascii-smuggling","tag-microsoft-security","tag-prompt-injection","tag-unicode"],"jetpack_featured_media_url":"https:\/\/www.0xprathamesh.com\/blog\/wp-content\/uploads\/2026\/09\/ascii-smuggling-unicode-nfkc.png","_links":{"self":[{"href":"https:\/\/www.0xprathamesh.com\/blog\/wp-json\/wp\/v2\/posts\/32","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.0xprathamesh.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.0xprathamesh.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/www.0xprathamesh.com\/blog\/wp-json\/wp\/v2\/comments?post=32"}],"version-history":[{"count":2,"href":"https:\/\/www.0xprathamesh.com\/blog\/wp-json\/wp\/v2\/posts\/32\/revisions"}],"predecessor-version":[{"id":94,"href":"https:\/\/www.0xprathamesh.com\/blog\/wp-json\/wp\/v2\/posts\/32\/revisions\/94"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.0xprathamesh.com\/blog\/wp-json\/wp\/v2\/media\/33"}],"wp:attachment":[{"href":"https:\/\/www.0xprathamesh.com\/blog\/wp-json\/wp\/v2\/media?parent=32"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.0xprathamesh.com\/blog\/wp-json\/wp\/v2\/categories?post=32"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.0xprathamesh.com\/blog\/wp-json\/wp\/v2\/tags?post=32"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}