But how is this possible?

[NoN']

Klaus, I don't want to be injust, even if the goal of my post wasn't to improved the scripts of the stack, congratulation for this amelioration !
So thank you very much for your post.
Best regards.

[bn]

Hi Renaud,

Here is a version of button Analyser that is pretty fast.

I tested against "War and Peace", Gutenberg 3.4 MB

Nombre de mots dans le texte : 566719
Nombre de mots extraits : 18575
Extraction réalisée en : 16 secondes

Also against

The Project Gutenberg
Du côté de chez Swann by Marcel Proust
1.1 MB
(in French)
Nombre de mots dans le texte : 169822
Nombre de mots extraits : 14381
Extraction réalisée en : 2 secondes

It would need some validation against short texts with all the possible appearences of a "word" to make sure ist does not omits any.

You might want to copy this button to your stack and test it for yourself.

Kind regards
Bernd

[NoN']

Thank you bernd for these modification proposals which I will adopt - for the most part - in all my next loops.

All the proposals made in this post are important and do indeed bring the processing time down to something acceptable.

But I still say that all this takes us away from the original question, which has not really been addressed.
As a reminder, my post wasn't aimed at improving script quality (it is not the good forum for that), but at understanding the reasons behind the difference in speed between different versions of Livecode. Unicode was mentioned as one of the probable causes of this problem. But is it the only one?

Bernd, you think your latest script is fast - and it is - but, with a small French text of 2822 words, it runs in less than a second* on Livecode 4.5 compared with 4 seconds on Livecode 10. (By the way, you didn't specify your hardware configuration, if it's not confidential).

So maybe I'm quibbling, but it seems to me that this isn't a subject to be treated lightly, because the perception of users, especially beginners, of processing speed remains crucial!

Renaud

* For Livecode 4.5 I had to disable the “split tSplitA with return as set” command in your script, which is not recognized in this version. But I don't think it had a crazy impact on execution time.

[bn]

Hi Renaud,

My impression is not that LC as a whole has slowed down. You are looking at a specific aspect of LC namely accessing text and processing it repeatetly.
I know that times to access "word x of line y" grow exponentially with the number of lines. And you do that very often in your script. But, as you point out, does not explain the slowdown between versions. Mark Waddingham repeatedly mentioned "unicode" as a reason for slowdown. But then unicode gives you the opportunity to use all languages of the world and not only a couple of Western languages. Also it lets you write from right to left.
It is not hard to imagine that the engine has to do a lot more work to accomodate for this.
But in the end only Mark Waddingham could give an informed answer.

As to my setup: I use a MacBook Pro with an M1 Max Arm processor. The arm processors are amazingly fast. 32 MB memory. System is MacOS Sequoia 15.5, LC 10.0.x.

I am amazed at the low speed you report for a relatively small text (2822 words).

I append here a version I think is fast and should run using LC 4.5.
Again copy the button to your stack and run it.

On a side note: When doing a "next repeat" I decrease the word count in variable comb, which is probably correct.

Could you please test this version and tell us the timings.
My aim is to run your Script against Du côté de chez Swann by Marcel Proust without going à la recherche du temps perdu

Kind regards
Bernd

[richmond62]

my post wasn't aimed at improving script quality (it is not the good forum for that)

I am not sure about that.

If this is not "the good forum for that", which is the good forum?

[NoN']

If this is not "the good forum for that", which is the good forum?

For exemple :
Developing With LiveCode
Talking LiveCode

Consider the title of the post is: "But how is this possible?" and not "Help me improve my poor code".

It seems that the interpretation of a question depends on :
- the terms chosen to ask the question (this is my responsibility, and I apologize if the question was unclear);
- our knowledge of the question asked;
- but also on the type of answer we want to give (for exemple your answer, which in no way answers the question posed in the initial post).

[richmond62]

(for exemple your answer, which in no way answers the question posed in the initial post)

1. No it doesn't; but it did ask a question about a trailing 'thing' you bunged in your post.

2. En Anglais vous ecrivez le mot 'exemple' avec un 'A': exAmple.

[richmond62]

And the MAIN 'answer' to your question sits with the implementation of complete Unicode compliance (earlier versions of LiveCode were partly unicode compliant, but one had to play around with surrogate pairs).

Of course this is a bit awkward with Macintosh as 'current' macOS systems (12-26) will not run pre-unicode versions: I am not sure how that sits with Windows: Linux does just fine.

[richmond62]

Pre LC version 7 the engine only had to "chew" (most of the time) through the 512 characters of the extended ASCII set: come unicode and the engine had to really "chew" through however many 100 thousand possible characters in the Unicode thing.

What amazes me is NOT that that is slower than just coping with the extended ASCII set, but that it is not actually far slower than it is. AMAZING!

[dunbarx]

Is it feasible to be able to disable the unicode portion of the engine? I do not really know what I am about to be talking about, but it sounds like unicode is a superset of ASCII.

[richmond62]

Is it feasible to be able to disable the unicode portion of the engine?

At one point the idea of modularisation was raised . . .

. . . as it has been raised 'elsewhere'.

But in both cases it seems to have come to nothing.
-

-
There is no obvious way to disable Unicode capability in the standalone builder.

[richmond62]

I assume the Unicode capability is baked into the engine.

[richmond62]

For what it is worth, Claude GPT sez:

Based on the search results, Unicode capabilities in LiveCode are built directly into the engine, not implemented as stacks within the IDE.

The robot's emphasis.

The Unicode support was a major engine refactoring effort that culminated in LiveCode 7.0. Unicode text required "such a monumental effort to get LiveCode 7.0 ready for release" precisely because it involved deep changes to the engine's text processing core.
Key evidence that this is engine-level implementation:

1. Core data files: The engine requires Unicode-related data files (icu-core.dat, icu-iter.dat and icu-coll.dat) to run, with the core data file being essential - "the engine will not run without it" What’s new | LiveCode
2. Internal encoding: The engine automatically handles UTF-16 encoding for all text operations and provides transcoding capabilities to other encodings
3. Backwards compatibility: The engine maintains automatic conversion between different platform encodings for legacy support, which would require engine-level implementation
4. Performance impact: Text manipulation performance was affected because it's "at the heart of the LiveCode engine" and "almost every command and function in LiveCode interacts" with the new Unicode processing LiveCode 7 | LiveCode

The deprecated functions like unicodeText, uniEncode, and uniDecode from pre-7.0 versions were also engine functions, not stack-based implementations, further confirming that Unicode handling has always been an engine responsibility in LiveCode.

https://livecode.com/livecode-7-0/

The beautiful thing about Unicode in LiveCode 7 is that you won’t notice it. We’ve spent 18 months pulling the engine apart and rewriting it, putting in 5 man years and changing 2000 files so that you simply won’t notice a thing. You can just use it. Anywhere. Fields, objects, buttons, user input, file paths… it all just works. Right to left text, Chinese, Russian, Spanish, French or hieroglyphics. LiveCode won’t bat an eyelid.

No: it won't bat an eyelid, but it will flutter its eyelashes more slowly than before: and because it is part of the engine Unicode stuff can not be excluded from a standalone.

[dunbarx]

Backwards compatibility: The engine maintains automatic conversion between different platform encodings for legacy support, which would require engine-level implementation

Hmmm. Isn't this, at heart, an issue of legacy support? I do not use unicode ever. I assert again I am not qualified to speak to this issue at all. But the above seems to indicate there is a "back door" to a non-unicode part of the engine. Not sure about the demand to do so.

Craig

[NoN']

2. En Anglais vous ecrivez le mot 'exemple' avec un 'A': exAmple.

When someone addresses me by making the effort to use a language that isn't their native language, I'm usually very forgiving of any linguistic errors they might make.
But this time, I understand that you might be outraged by my mistake. Indeed, this change of letter, the diabolical interchanging of the “a” with the “e” is such as to totally change the understanding of the word itself and therefore of the sentence as a whole!

So thank you for rectifying my error and demonstrating your vigilance and greatness of spirit, and I hope you'll excuse any further errors in this post itself...

Apart from that, thank you very much for your detailed explanations. This is really the type of information I was looking for initially.

Renaud