PHP unexpected output when replacing character code 8217 in string

Question

I'm running into an unexpected character replacement problem. The character code is 8217, ’.

I've tried escaping the character with a slash, but it didn't make a difference.

php > $a = preg_replace('/([.,\'"’:?!])<\/a>/', '</a>\1', 'letter">Evolution’</a> </li>');
php > echo($a);
// => letter">Evolution/a> </li>

// Just to show that it works if the character is different
php > $a = preg_replace('/([.,\'"’:?!])<\/a>/', '</a>\1', 'letter">Evolution"</a> </li>');
php > echo($a);
letter">Evolution</a>" </li>

I would expect it to output

letter">Evolution</a>’ </li>

instead of

letter">Evolution/a> </li>

Answer 1

By default, pcre (the php regex engine) considers your pattern as a succession of single byte encoded characters. So when you write [’] you obtain a character class with the three bytes on which THE RIGHT SINGLE QUOTATION MARK (U+2019) is encoded, i.e.: \xE2, \x80, \x99.

In other words, writting "/[’]/" in this default mode is like writting "/[\xE2\x80\x99]/" or "/[\x80\xE2\x99]/" or "/[\x99\xE2\x80]/" etc., the regex engine doesn't see a sequence of bytes that represents the character ’ but only three bytes.

This is the reason why you obtain a strange result, because [.,\'"’:?!] will only match the last byte of ’ so \x99.

To solve the problem, you have to force the regex engine to read your pattern as an UTF-8 encoded string. You can do that with one of this ways:

• preg_replace('~(*UTF)([.,\'"’:?!])</a>~', '</a>\1', 'letter">Evolution’</a> </li>');
• preg_replace('~([.,\'"’:?!])</a>~u', '</a>\1', 'letter">Evolution’</a> </li>');

This time the three bytes \xE2\x80\x99 are seen as an atomic sequence for the character ’.

Notice: (*UTF) is only for the reading of the pattern but the u modifier does more things: it extends shorthand character classes (like \s, \w,\d) to unicode characters and checks if the subject string is utf-8 encoded.

Answer 2

Just add unicode flag to the regex:

$a = preg_replace('/([.,\'"’:?!])<\/a>/u', '</a>\1', 'letter">Evolution’</a> </li>');
#                              here ___^
echo($a);