unidecode(uniencode()) removes characters??

[hliljegren]

If I do a:

Code: Select all

put unidecode(uniencode("åäö are som swedish characters", "utf8"))

the result is:

Code: Select all

 are som swedish characters

The swedish characters are stripped from the text! Am I missing something or what?

If I strip the "utf8" part (which converts to UTF-16 if I understand things correctly) everything works fine, but I really need to convert to UTF-8 as my database stores all fields in that format.

Should I file a bug towards runrev or towards my own knowledge?

[Mark]

hliljegren,

Your script should be like this:

// convert to RunRev unicode (UTF16)
put uniencode("åäö") into myUnicode
// show unicode in field
set the unicodeText of fld 1 to myUnicode
// convert to UTF8
put unidecode(myUnicode,"UTF8") into myUTF8

What you get in myUTF8 is binary data. You can't put that into a field, but you can write it to a file, or your database, and tell a text editor to open it as UTF8.

Best regards,

Mark

[Mark Smith]

I think you need to move the "utf8" declaration into the unidecode call:

Code: Select all

unidecode(uniencode("åäö are som swedish characters"),"UTF8")

The way you had it will translate a string from utf8 to whatever your system's encoding is.

Best,

Mark Smith

[hliljegren]

Aha! Thanks!

If I understand your last post correct I can't translate a string directly to UTF-8 via uniencode. Instead uniencode ALWAYS translates to UTF-16.

So uniencode(myText, utf8) translates a UTF-8 encoded into UTF-16 and thus to get a string from a field to utf-8 format I need to encode it to UTF-16 and then "decode" it to UTF-8 which I then can send to my MySQL database.

Correct?

[Mark]

Hi Mark,

From the docs:

If you don't specify a language, the uniDecode function returns the stringToDecode, with every second byte removed.

As far as I can see, this is independent from any encoding.

Best,

Mark