Text Problems over Socket Connection

[Klaus]

Oh, you replied while I was replying

Well, I am no socket expert, but it sound like they only like ASCII somehow.
As proposed earlier in this thread, you could base64encode your UTF8 string
so it gets "converted" to ASCII.

No idea this will work, but at least worth a try

[aircooled76]

Yeah... I figured that out eventually... other problems now... one solved next one crops up!!! (see my more recent post)

[aircooled76]

The commands I need to send over the socket need to be UTF8... this is good as I need special Charactors (ANSII is no good)

The problem is I have a whole lot of ANSII text in LiveCode along with the one field that needs to be UTF8 or 16... Need to figure a way how to get it all to pass through nicely... Ill have to have a crack tomorrow... it is getting late over here... Hopefully there can be some more suggestions I can test out in the meantime!

[Klaus]

Sorry, no more ideas...

[aircooled76]

Just noticed something interesting...

when I use the code:

Code: Select all

on mouseUp
   set the unicodeText of field "testfield2" to the unicodeText of field "testfield"
   put the unicodetext of fld "testfield2" into tData
   put uniDecode(tData,"utf16") into field "testfield3"
end mouseUp

I get "Č Ž Š Đ Ć" in testfield2

but only get "C Ž Š Ð C" in testfield3... the "Č" and the "Ć" just turn into "C"

There must be a solution for this... or do I say to my Croatian friend... "sorry livecode won't play nicely with your countries names"

[Klaus]

Hi,

just made a test with a couple of fields and this seems to do the job:

Code: Select all

on mouseup
   ## Some chinese text in fld 1, works, same strange characters in fld 2 :-D
   set the unicodetext of fld 2 to the unicodetext of fld 1

   ## put unicodetext into variable
   put the unicodetext of fld 2 into tt

   ## as mentioned earlier, "turn" binary(sic!) unicodetext to ASCII
   put base64encode(tt) into tt2
   
   ## Test: Yes, really two lines of pure ASCII :-)
   put tt2

   ## When "converted" back to unicode, this line works as exspected:
   set the unicodetext of fld 3 to base64decode(tt2)
end mouseup

Best

Klaus

[FourthWorld]

People sometimes say it's overkill, but I've found time and again that when I have transport issues with binary data like Unicode, converting it to base64 seems to take care of it.

There may be ways to adjust the header to address that without converting to base64, but base64 is so convenient in LC that it's often a pretty good way to resolve such issues.

[BvG]

People sometimes say it's overkill, but I've found time and again that when I have transport issues with binary data like Unicode, converting it to base64 seems to take care of it.

There may be ways to adjust the header to address that without converting to base64, but base64 is so convenient in LC that it's often a pretty good way to resolve such issues.

completely not the problem, please read threads more carefully

[FourthWorld]

completely not the problem, please read threads more carefully

My apologies. With so many posts and my volunteer admin tasks here, I rarely have the time to completely re-read every thread here before replying, and admittedly this may sometimes mean a post from me that doesn't account for errors in other posts.

So if I understand you correctly, the base64Encode included in so many prior posts is completely unnecessary?

[BvG]

So if I understand you correctly, the base64Encode included in so many prior posts is completely unnecessary?

that too of course, but actually the problems Aircooled has are more in the direction of basic programming, like how to use functions and repeat loops. The somewhat weird way how LC does Unicode doesn't help either (Great job Klaus, in trying to continuingly offering help btw).

[aircooled76]

OK... so I got a bit frustrated and went onto other projects... but now I am back to slay this Unicode thing... (hopefully just in time for v7 of LiveCode to be released with a code-wide fix!

I digress...

My current attempt to fix this is this:

Code: Select all

write "CG 1-" & field "coach3rdvl" of card settings && "add" && field coach3rdfl of card settings &&quote& field coach3rd of the card settings &"_home" &quote&" 1 "&quote&"<templateData><componentData id=\"&quote&"_name\"&quote&">" & cr & \
"<data id=\"&quote&"text\"&quote&" value=\" &quote& uniDecode (the unicodetext of field T1Coach,"UTF8")  & "\"&quote&">" & cr & \
"</data> </componentData><componentData id=\"&quote&"_club\"&quote&">" & cr & \
"<data id=\"&quote&"text\"&quote&" value=\" &quote& field t1fullname of card Controller & "\"&quote&">" & cr & \
      "</data></componentData></templateData>"&quote&"" & format("\r\n") to socket field IP of card settings

The field T1Coach has " ščćđž ČĆŽŠĐ" in it but what comes out in the destination program is just ????? but what gets to the graphics is šžŽŠ. No sign of the čćđČĆĐ. I have tried using

Code: Select all

uniDecode (the unicodetext of field T1Coach,"UTF16")

but that just gives me " ?ccd? CC???" in the destination program or "ccd CC" or in the graphics.

I feel like I am bumbling along in the dark... if anyone can shed some light this way I would appreciate it!

[bangkok]

You might use base64encode, no ?

Code: Select all

Use the base64Encode function to encode binary data for transmission over communication channels that don't accept raw binary data.

[aircooled76]

OK... I did a bit more testing...

I downloaded a SocketTest and tuned into the port LiveCode is sending to http://sockettest.sourceforge.net/

When i sent

Code: Select all

uniDecode (the unicodetext of field T1Coach,"UTF8") 

I get at SocketTest:

Code: Select all

CG 1-2 add 4 "coach_home" 1 "<templateData><componentData id=\"_name\">
<data id=\"text\" value=\" š�ćđž ČĆŽŠ�\">
</data> </componentData><componentData id=\"_club\">
<data id=\"text\" value=\"HOME TEAM NAME\">
</data></componentData></templateData>"

When i sent

Code: Select all

uniDecode (the unicodetext of field T1Coach,"UTF16") 

I get at SocketTest:

Code: Select all

CG 1-2 add 4 "coach_home" 1 "<templateData><componentData id=\"_name\">
<data id=\"text\" value=\" šccdž CCŽŠÐ\">
</data> </componentData><componentData id=\"_club\">
<data id=\"text\" value=\"HOME TEAM NAME\">
</data></componentData></templateData>"

I also sent the following code to save to a text file:

Code: Select all

on mouseUp
   ask file "Select a location to save to"
    if the result is not "cancel" then
put "CG 1-" & field "coach3rdvl" of card settings && "add" && field coach3rdfl of card settings &&quote& field coach3rd of the card settings &"_home" &quote&" 1 "&quote&"<templateData><componentData id=\"&quote&"_name\"&quote&">" & cr & \
"<data id=\"&quote&"text\"&quote&" value=\" &quote& uniDecode (the unicodetext of field T1Coach,"UTF8")  & "\"&quote&">" & cr & \
"</data> </componentData><componentData id=\"&quote&"_club\"&quote&">" & cr & \
"<data id=\"&quote&"text\"&quote&" value=\" &quote& field t1fullname of card Controller & "\"&quote&">" & cr & \
      "</data></componentData></templateData>"&quote&"" & format("\r\n") into url ("file:" & it) 
end if
end mouseUp

And that resulted in:

Code: Select all

CG 1-2 add 4 "coach_home" 1 "<templateData><componentData id=\"_name\">
<data id=\"text\" value=\" ščćđž ČĆŽŠĐ\">
</data> </componentData><componentData id=\"_club\">
<data id=\"text\" value=\"HOME TEAM NAME\">
</data></componentData></templateData>"

That seems to be correct (checked in NotePad++ and it displays correctly in UTF8

On reading CasparCG's (the destination software) website all data is assumed to be UTF8...

I have more information but still feel in the dark!

I havn't tried base64encode yet... but I seemed to have found the problem... the font in the graphics templates does not seem to accept all the Characters I am sending... I tested with another graphics template (with Arial font) and all the characters went through OK...

Big shout out to Monte from MergEXT who gave me a tip on email... cheers guys...

[visionmill]

Hi

I am looking to do exactly the same thing with CasparCG. Did you ever get a way of generating the UTF8 correctly?

[aircooled76]

Yeah... go it working.. More info on CasparCG forum here: http://casparcg.com/forum/viewtopic.php?f=9&t=1467