Ill-coded identifier crashes python when coding spec is utf-8

[hyeshik]

BPO	1037
Nosy	@gvanrossum, @hyeshik

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2007-08-29.18:55:13.844>
created_at = <Date 2007-08-27.12:43:24.965>
labels = ['expert-unicode', 'type-crash']
title = 'Ill-coded identifier crashes python when coding spec is utf-8'
updated_at = <Date 2008-01-06.22:29:45.648>
user = 'https://github.com/hyeshik'

bugs.python.org fields:

activity = <Date 2008-01-06.22:29:45.648>
actor = 'admin'
assignee = 'none'
closed = True
closed_date = <Date 2007-08-29.18:55:13.844>
closer = 'gvanrossum'
components = ['Unicode']
creation = <Date 2007-08-27.12:43:24.965>
creator = 'hyeshik.chang'
dependencies = []
files = []
hgrepos = []
issue_num = 1037
keywords = []
message_count = 2.0
messages = ['55335', '55432']
nosy_count = 2.0
nosy_names = ['gvanrossum', 'hyeshik.chang']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue1037'
versions = ['Python 3.0']

[hyeshik]

Illegal identifier makes python crash on UTF-8 source codes/interpreters.

Python 3.0x (py3k:57555M, Aug 27 2007, 21:23:47) 
[GCC 3.4.6 [FreeBSD] 20060305] on freebsd6
>>> compile(b'#coding:utf-8\n\xfc', '', 'exec')
zsh: segmentation fault (core dumped)  ./python

The problem is that tokenizer.c:verify_identifer doesn't check
return value from PyUnicode_DecodeUTF8 but some invalid utf8
sequences could be there.

[gvanrossum]

r57674.
I'm not sure this is the right fix though...