repeat for each token and the character '\'

[rinzwind]

The repeat loop exits as soon as it encounters a \ ... seems like an ugly bug to me?
Or am I missing some hidden feature?

put "1 2 3 4 5 \ 6 7 8 9 10" into test
repeat for each token t in test
put t & comma after msg
end repeat

[rinzwind]

or -- or //
seems to quit on some live code script constructs

[rinzwind]

any workaround? i was using token as this most closely resembles what I'm looking for...
word construct includes characters like (test
maybe customizable delimiters?
I hoped one could specify the tokens that 'token' sees as a delimiter... would be way more flexible.

[Martin Koob]

The "\" character is used to break a line in a script so you can have one long line of script shown on multiple shorter lines. Inside a quoted string it does not have that effect.

You are right it seems when tries to set t= "\" it exits the repeat loop. Perhaps the fact that this character is interpreted as a script line break is causing this somehow.

Martin

[Martin Koob]

You can set the itemDelimiter to a specific character and then repeat for each item.

Martin

[Martin Koob]

or -- or //
seems to quit on some live code script constructs

'--' and '//' are used to comment out lines of script. So they are also characters that affect the parsing lines of script just like '\' . Another set of characters that does that is '/*' which is used with a following "*/" for commenting out blocks of code. Putting "/*" in your text variable will also cause the exit from the repeat loop.

So using any of these character sequences that are used for parsing scripts causes a problem. Not sure if it is a bug or if there is a workaround if you needed to use these characters as tokens.

Martin

[FourthWorld]

Because "\" is within a quoted string I would consider this a bug. Please feel free to file a bug report against this:
http://quality.runrev.com/

[Martin Koob]

It is interesting because when quoted in the string the '\' character (or '--' or '//' or '/*' does not pose a problem, that line executes normally.

It is when that '\' character is used by the repeat loop as one of the tokens in whatever data structure holds 'every token' that the issue occurs.

Martin

[Martin Koob]

Furthermore, this does not cause a problem when you use each 'item'.

If I make the string a comma separated list and repeat for each item there is not a problem.

put "1,2,3,4,5,\,6,7,8,9,10" into test
repeat for each item t in test
put t & comma after msg
end repeat

Nor does it affect each 'word'

put "1 2 3 4 5 \ 6 7 8 9 10" into test
repeat for each word t in test
put t & comma after msg
end repeat

Martin

[FourthWorld]

After reading Martin's last two posts I think he's onto something and that this isn't a bug after all. I had overlooked that the chunk type used is "token", which is very specialized and highly dependent on specifics within the deep innards of how the engine expects to encounter strings. Since all the elements are already space-delimited, the "word" chunk type is much simpler to use here.

[LCMark]

@FourthWorld: You are correct - the 'token' chunk is tied to the notion of 'token' used internally by the script parser. In this case '\' is the 'next line continuation character' meaning - ignore all characters after this until the next newline and then continue processing after that. Since there is no newline in 'test', in this case it skips everything after '\' and finishes iterating.

[livecodeali]

The idea of the token chunk (at least in its current hard-coded form) is that a token is a significant chunk with respect to parsing the LiveCode script language. Therefore for any of the following, you will see the number of tokens = 0, token n = empty:

"/* some text */"
"// some text"
"-- some text"
"# some text"

since these are all comments in LiveCode script and therefore not significant chunks. It is not that the repeat loop is cause to exit as such, merely that there are no further tokens so the repeat loop is complete.

In script, "\" is the line continuation character, and thus everything else on the current line is disregarded as far as the token chunk is concerned. The next token would be the first token of the next line. The following

put "1 2 3 4 5 \" & return & " 6 7 8 9 10" into test
repeat for each token t in test
put t & comma after msg
end repeat

would give you what you expected.

At some point we intend to introduce a configurable version of the token chunk, so that things can be tokenized in any way you like, but for now you're stuck with exactly how the LiveCode script parser interprets a token.

[livecodeali]

Heh, pipped by @LCMark