Normalizing text

[hopkins]

Hello,

Is there a way to convert text with accentuation into a "normalized" text ?

For example: transform "Béla Bartók" into "Bela Bartok"

I cannot seem to get the function "normalize" (https://livecode.fandom.com/wiki/NormalizeText) to work, but I may not be using it correctly.

[jmburnod]

Hi,
I tried to work first time with normalizeText function but i dont understand yet.
I use this for similar goal:

Code: Select all

function fromAccentToNot pText
   put "àâäçéèêëîïòôùû" into tAccent
   put "aaaceeeeiioouu" into tNoAccent
   put 0 into tCount
   repeat for each char tChar in tAccent
      add 1 to tCount
      replace tChar with char tCount of tNoAccent in pText
   end repeat
   return  pText
end fromAccentToNott

Best regards
Jean-Marc

[jacque]

Normalize isn't the right term. It only works with a few characters that are visually identical but which have different unicode values. It isn't meant to remove diacriticals, those are actually part of the character itself.

I think Jean-Marc's method is all you can do.

[[-hh]]

@Jean-Marc
Two considerations.
1. One should possibly use the casesensitive.
2. Sometimes 'accented' chars are replaced by two chars, for example "Ä" with "Ae" or "ß" with "sz".

Here is a mapping that I once used (=fld "Mapping" in the code below).

Code: Select all

Á to A
Á to A
Ä to Ae
 to A
À to A
à to A
Å to A
Č to C
Ç to C
Ć to C
Ď to D
É to E
Ě to E
Ë to E
È to E
Ê to E
Ẽ to E
Ĕ to E
Ȇ to E
Í to I
Ì to I
Î to I
Ï to I
Ň to N
Ñ to N
Ó to O
Ö to Oe
Ò to O
Ô to O
Õ to O
Ø to O
Ř to R
Ŕ to R
Š to S
Ť to T
Ú to U
Ů to U
Ü to Ue
Ù to U
Û to U
Ý to Y
Ÿ to Y
Ž to Z
á to a
ä to ae
â to a
à to a
ã to a
å to a
č to c
ç to c
ć to c
ď to d
é to e
ě to e
ë to e
è to e
ê to e
ẽ to e
ĕ to e
ȇ to e
í to i
ì to i
î to i
ï to i
ň to n
ñ to n
ó to o
ö to oe
ò to o
ô to o
õ to o
ø to o
ð to o
ř to r
ŕ to r
š to s
ť to t
ú to u
ů to u
ü to ue
ù to u
û to u
ý to y
ÿ to y
ž to z
þ to b
Þ to B
Đ to D
đ to d
ß to sz
Æ to AE
Πto OE
æ to ae
œ to oe

Then the "Replace" button code is alike yours:

Code: Select all

on mouseUp
  lock screen; lock messages
  put fld "TextIn" into s
  put fld "Mapping" into m
  replace " to " with comma in m
  set the casesensitive to true
  repeat for each line L in m
    replace (item 1 of L) with (item 2 of L) in s
  end repeat
  put s into fld "TextOut"
end mouseUp

[hopkins]

Thank you for your answers. I wish there were a simpler way of doing this, but the above code will certainly do the job.

[[-hh]]

Possibly not "simpler" but for long input strings and replacement mappings 20-30 times faster by using regular expressions (btn "ReplaceText"):

Code: Select all

on mouseUp
  put the millisecs into m1
  lock screen; lock messages
  put fld "TextIn" into s
  put length(s) into n
  put fld "Mapping2" into m2
  replace " to " with comma in m2
  repeat for each line L in m2
    put replaceText(s,item 1 of L,item 2 of L) into s
  end repeat
  put s into fld "TextOut2"
  put n & ": " & (the millisecs - m1) && (s is fld "TextOut") into fld "timing"
end mouseUp

The replacement mapping (fld "Mapping2"):

Code: Select all

Á|Á|Â|À|Ã|Å to A
Þ to B
Č|Ç|Ć to C
Ď|Đ to D
É|Ě|Ë|È|Ê|Ẽ|Ĕ|Ȇ to E
Í|Ì|Î|Ï to I
Ň|Ñ to N
Ó|Ò|Ô|Õ|Ø to O
Ř|Ŕ to R
Š to S
Ť to T
Ú|Ů|Ù|Û to U
Ý|Ÿ to Y
Ž to Z
á|â|à|ã|å to a
þ to b
č|ç|ć to c
ď|đ to d
é|ě|ë|è|ê|ẽ|ĕ|ȇ to e
í|ì|î|ï to i
ň|ñ to n
ó|ò|ô|õ|ø|ð to o
ř|ŕ to r
š to s
ť to t
ú|ů|ù|û to u
ý|ÿ to y
ž to z
Æ to AE
Ä to Ae
Πto OE
Ö to Oe
Ü to Ue
ä|æ to ae
ö|œ to oe
ß to sz
ü to ue

[hopkins]

Cool, thanks

[jmburnod]

Hi,
@Hermann

Two considerations...

Yes, you're right
Thank you for doing the job
Jean-Marc

[richmond62]

"ß" with "sz"

That's odd:

Straße = Strasse