"Run Selection/Line in Python REPL" hangs up if the code contains non-ascii characters

[tomoki]

Type: Bug

Behaviour

When we select the following line in py file and "Run Selection/Line in Python REPL" , the Python REPL shows the loading icon forever and doesn't return.

print("こんにちは世界")

We see the following error in the developer tool.

[Extension Host] Log: Traceback (most recent call last):
  File "/home/tomoki/.vscode/extensions/ms-python.python-2024.20.0-linux-x64/python_files/python_server.py", line 183, in <module>
    request_json = json.loads(request_text)
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/json/decoder.py", line 341, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 111 (char 110)

There is a mismatch between "Content-Length" and its content, and it tries to read the next header and causes the failure.

Steps to reproduce:

1. Create a.py
2. Write print("こんにちは世界")
3. Right-click and select "Run Python" -> "Run Selection/Line in Python REPL"

Actual result: We see the loading icon in Python REPL, and it doesn't return.

Expected result: We see the execution result "こんにちは世界".

Note that the other non-ascii characters can cause the issues. (For example, print("Olá Mundo!"))

Diagnostic data

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

2024-11-22 21:42:42.619 [info] Experiment 'pythonRecommendTensorboardExt' is active
2024-11-22 21:42:42.619 [info] Experiment 'pythonTerminalEnvVarActivation' is active
2024-11-22 21:42:42.619 [info] Experiment 'pythonTestAdapter' is active
2024-11-22 21:42:42.619 [info] Native locator: Refresh started
2024-11-22 21:42:42.619 [info] Native locator: Refresh started
2024-11-22 21:42:42.630 [info] > pyenv which python
2024-11-22 21:42:42.630 [info] cwd: .
2024-11-22 21:42:42.913 [info] Native locator: Refresh finished in 340 ms
2024-11-22 21:42:42.916 [info] Python interpreter path: /bin/python3
2024-11-22 21:42:42.917 [info] Editor support is inactive since language server is set to None.
2024-11-22 21:42:50.866 [info] > /bin/python3 ~/.vscode/extensions/ms-python.python-2024.20.0-linux-x64/python_files/normalizeSelection.py

Extension version: 2024.20.0
VS Code version: Code 1.95.3 (f1a4fb101478ce6ec82fe9627c43efbf9e98c813, 2024-11-13T14:50:04.152Z)
OS version: Linux x64 6.11.7-amd64
Modes:

• Python version (& distribution if applicable, e.g. Anaconda): 3.12.7
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): System
• Value of the python.languageServer setting: None

User Settings

languageServer: "None"

Installed Extensions

Extension Name	Extension Id	Version
C/C++	ms-vscode.cpptools	1.22.11
C/C++ Extension Pack	ms-vscode.cpptools-extension-pack	1.3.0
C/C++ Themes	ms-vscode.cpptools-themes	2.0.0
Clang-Format	xaver.clang-format	1.9.0
CMake	twxs.cmake	0.0.17
CMake Tools	ms-vscode.cmake-tools	1.19.52
Docker	ms-azuretools.vscode-docker	1.29.3
EditorConfig for VS Code	EditorConfig.EditorConfig	0.16.4
JavaScript Debugger	ms-vscode.js-debug	1.95.3
JavaScript Debugger Companion Extension	ms-vscode.js-debug-companion	1.1.3
Pylance	ms-python.vscode-pylance	2024.11.3
Python	ms-python.python	2024.20.0
Python Debugger	ms-python.debugpy	2024.12.0
Remote - SSH	ms-vscode-remote.remote-ssh	0.115.1
Remote - SSH: Editing Configuration Files	ms-vscode-remote.remote-ssh-edit	0.87.0
Remote Explorer	ms-vscode.remote-explorer	0.4.3
Scala (Metals)	scalameta.metals	1.43.0
Scala Syntax (official)	scala-lang.scala	0.5.8
Table Visualizer for JavaScript Profiles	ms-vscode.vscode-js-profile-table	1.0.10
Trailing Spaces	shardulm94.trailing-spaces	0.4.1

System Info

Item	Value
CPUs	AMD Ryzen 9 8945HS w/ Radeon 780M Graphics (16 x 1097)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: enabled_on direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_graphite: disabled_off video_decode: enabled video_encode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off webnn: disabled_off
Load (avg)	0, 0, 0
Memory (System)	29.16GB (21.27GB free)
Process Argv	--crash-reporter-id c7a91091-dcd6-4304-a0c4-f75dff2224f9
Screen Reader	no
VM	0%
DESKTOP_SESSION	lightdm-xsession
XDG_CURRENT_DESKTOP	XFCE
XDG_SESSION_DESKTOP	lightdm-xsession
XDG_SESSION_TYPE	x11

A/B Experiments

vsliv368cf:30146710
vspor879:30202332
vspor708:30202333
vspor363:30204092
vscod805:30301674
binariesv615:30325510
vsaa593cf:30376535
py29gd2263:31024239
vscaat:30438848
c4g48928:30535728
azure-dev_surveyone:30548225
vscrpc:30673769
962ge761:30959799
pythonnoceb:30805159
asynctok:30898717
pythonmypyd1:30879173
h48ei257:31000450
pythontbext0:30879054
cppperfnew:31000557
dsvsc020:30976470
pythonait:31006305
dsvsc021:30996838
724cj586:31013169
dvdeprecation:31068756
dwnewjupytercf:31046870
nativerepl1:31139838
pythonrstrctxt:31112756
nativeloc1:31185841
cf971741:31144450
iacca1:31171482
notype1:31157159
5fd0e150:31155592
dwcopilot:31170013
stablechunks:31184530

[tomoki]

I'll create a PR to fix the issue and will link to this issue.

Update: I opened #24480

[tomoki]

I confirmed that we see the same issue at macOS.

Extension version: 2024.20.0
VS Code version: Code 1.95.3 (Universal) (f1a4fb101478ce6ec82fe9627c43efbf9e98c813, 2024-11-13T14:50:04.152Z)
OS version: Darwin arm64 24.1.0
Modes:

-   Python version (& distribution if applicable, e.g. Anaconda): 3.13.0
-   Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Global
-   Value of the `python.languageServer` setting: None

[karthiknadig]

Thank for the detailed report and the fix for this.