PHP's Problem with Character Encoding

The basic problem PHP has with character encoding is it has a very simple idea of what the notion of a character is: that one character equals one byte. Being more precise, the problem is most of PHP‘s string related functionality (see common_problem_areas_with_utf-8 for further details) make this assumption but to be able to support a wide range of characters (or all characters, ever, as Unicode does), you need more than one byte to represent a character.

An example in code. From Sam Ruby’s i18n Survival Guide, he recommends using the string Iñtërnâtiônàlizætiøn for testing. Counted with your eye, you can see it contains 20 characters;

Iñtërnâtiônàlizætiøn
12345678901234567890

But counted with PHP‘s strlen function...

<?php
echo strlen('Iñtërnâtiônàlizætiøn');
?>

PHP will report 27 characters. That’s because the string, encoded as UTF-8, contains multi-byte characters which PHP‘s strlen function will count as being multiple characters.

Life gets even more interesting if you run the following2);

<?php
header('Content-Type: text/plain; charset=ISO-8859-1');

$str = 'Iñtërnâtiônàlizætiøn';

$out = '';
$pos = '';
for($i = 0, $j = 1; $i < strlen($str); $i++, $j++) {
$out .= $str[$i];
if ( $j == 10 ) $j = 0;
$pos .= $j;
}

echo $out."\n".$pos;
?>

You should see something like;

Iñtërnâtiônà lizætiøn
123456789012345678901234567