Filter without empty misses empty last line(s)

[edgore]

I have run into this many times before, but since it keeps coming up, I figured I would ask about it, since I am probably doing something wrong.

"filter variable without empty" will get rid of all the empty lines within a variable, except for, sometimes the last line, or, other times, the last several lines (all empty). I get around this using something like:

repeat forever
if the last char of variable is return then
delete the last char of variable
else
exit repeat
end if
end repeat

Which seems like a silly thing to do.

Since that *does* work, though, it must mean that the last line(s) *was/were* actually empty -otherwise just repeating to remove returns would not get rid of them - that loop would not delete a line if there were some stray invisible character in the line. Yet, in every case I end up with no empty lines at the end of variable after running it

So, why is "filter variable without empty" not getting rid of them on its own?

Edit: I have added an attachment - a stack that shows what I am talking about. The left field (source) contains empty lines, and three empty lines at the end of the field. Click the button, it filters and puts it in the right field. All the empty lines are gone except for the last line.

[dunbarx]

I tried this with a field that had:

aa
bb

cc

and where line 4 ("cc") was the last line (total 4 lines. I then added a blank line below that line (total 5 lines)

I filtered the text and appended an "X" to the result. The return in the last line is not filtered, though the return in the third line is. In other words for the first case you get:

aa
bb
ccX

whereas in the second you get:

aa
bb
cc
X

Makes no difference how many empty lines follow the text. All but the last line will be filtered, The return in the penultimate line is there because even an empty last line requires it. I see this as normal.

Craig Newman

[edgore]

I agree that it's normal - it's the way that it has always worked. However, does it make sense? If I say filter without empty then I would expect every empty line to be removed from the container, rather than always having to test to see if the last line is empty or not.

If I was writing a function to remove empty lines I would have it check to see if there was a return at the end of the container creating a empty line, so it seems strange that the filter command doesn't do this.

Edit - and I understand what you are saying about the append - but if I have filtered a container to remove all empty lines I would expect my append to have to be "put return & "X" after variable", because that is how it works in every other case where I want to append something on a new line.

[dunbarx]

I sort of see what you mean, but if you have:

aa
bb
cc

and there is an empty line below the "cc, then what you really have is a line "cc" and a return character. That line does not get filtered. so the return lurks there. My experiment only shows that when I append something to the filtered text, the return does its thing.

Make two fields and a button. Put "aa" & return & "bb" & return & "cc" into fld 1. Put this into the button script:

Code: Select all

on mouseUp
   put "" into fld 2
   put the number of lines of fld 1 into temp
   get fld 1
   filter it without empty
   put it  && the number of lines of it  && temp into fld 2
end mouseUp

Now put a return after the "cc". Click again. It isn't that the last empty line is being missed, it is that if the last return char is present, it is available to do its thing.

Craig

[edgore]

Yeah, I guess I will always just have to do a "if the last char of variable is return then delete the last char of variable', because really what filter without empty does is the equivalent of "replace return & return with return", which makes sense in most cases. I run into problems because the downstream processing that happens with my stuff doesn't expect the final line of data to have a return at the end, and that is probably relatively unusual. You are right though - Livecode doesn't interpret the return at the end of the line as a new empty line, even though it looks like one in variable watcher and fields.

Thanks for helping me work through this! I figured that it was a problem with how I was thinking about it!

[SparkOut]

This is interesting because it's highlighted a new interpretation to me. For me, I've always thought of the return as the terminator at the end of each line, not as the beginning of a new line. Pretend that the # is a carriage return character, so in the list:

AA#
#
BB#
CC#

there is obviously an empty line between AA and BB. Filtering without empty will remove it, but just as AA# isn't filtered, nor is CC# even though it renders as

AA
#
#BB
#CC
#

with a trailing # on an empty line.

I don't feel smug, but at least I'm glad I've had what appears to be the easier of the ways to perceive things up to now. This has opened my eyes to other ways of looking at it. Why it should be apparent to me that it's a line terminator, yet if I count the number of chars of line AA# and get 2 and it doesn't bother me either, is now starting to get intriguing.

[edgore]

The thing to remember (and I knew this in the back of my head already, but I am frequently dumb unless I stop and think things through) is that what you are seeing displayed in a field is the location of *the insertion point* - the place where the next character typed will appear. So the insertion point is "on the other side" of the return, or on the next line, which doesn't exist.

The way you are thinking about it is correct (on the other hand, the way my stupid downstream mainframe program wants stuff formatted is stupid in ways I cannot describe).

There is definitely confusing behavior around returns though. For example, if you script "delete the last char" of a line, it doesn't delete the return, it deletes the last visible character, but if you script "delete the last char" of the the container, it deletes the return. (I have attached a stack that makes what I am saying clear).

So, in this case, it's not treating the return as part of the line. I'm going to claim this is the root of my confusion.

[dunbarx]

If you put "123" into fld 1:

and click below the text, the insertion point will place at the end of the line.

If you put "123" & return into fld 1

and click below the text, the insertion point will place at the next line (line 4). If you:

put "123" & return into fld 1
delete the last char of fld 1

and click below the text, the insertion point will place at the end of the line (line 1). You can count the chars in any of these cases, and see that the return char is a real char, that the text looks the same in each case, but that the two cases are quite different. Again, all this is expected behavior.

Craig

[edgore]

What I was referring to is that the return char is not counted as one of the chars of a line - and that this is the behavior that you would want, because it's a delimiter and when you ask for the number of characters in a line you almost always want to know how many visible characters there are. The return *is* counted when you get the number of chars in the entire field. I have attached an updated stack that makes it more clear what I am talking about - the differences that you get in the count that is what I am saying could potentially be confusing. For example, click the "delete last char of line 1" button four times and tell me that's a not a little weird if you don't think about it. The return on line 1 will never be deleted because, technically, it's not on line 1 - it's between line 1 and line 2 (which is just an insertion point really, not a line)

At this point though I am just sort of doing the equivalent of staring at my hand and going "Wow - dude, have you ever really *looked* at your hand?!?". I have been using Livecode for ages (and hypercard, supercard and toolbook before that, which I am sure work the same way) and understand how returns work and the expected behaviors, but until you pointed out what was going on with filter empty, I never really stopped to think about all the implications of return and the occasionally strange results that you can get if you change the scope of what you are looking at (line vs field, for example). They are things that make sense, once you think about them, but you have to think about them to see why they work they way they do in various situations.

Oh, and I figured out why I was sometimes seeming to get multiple blank lines at the end of something after filtering - it was my fault, of course. I had several containers that I filtered without empty, then appended them into one container. If any of them contained only a return then when they got appended I would get several stray lines at the end. Since I thought that the earlier filters were removing *all* empty lines (that a container that contained only a return would be filtered to empty), I was baffled by the end result, and panicked, figuring the safest thing was to recursively check for empty trailing returns. Based on your earlier help, I now know that I just have to filter on last time and check to see if there is a single return at the end, then delete it.

[dunbarx]

Hi.

All good therapy and experience.

But what makes you think that if you put "123" & return into fld 1, that there are only three chars in that field?

I had mentioned this in an earlier post, where I wanted you to count the chars in the scenarios I put forward.

Craig

[edgore]

I don't think that - I know that there are 4 chars in the field.

What I was saying was:

If you ask livecode for the number of lines in field 1 it would answer "1"
If you ask for the number of chars in that line it would answer "3"

So one could draw the conclusion that there are three characters in field 1 (since there is only 1 line, and it has 3 characters in it)

But, of course, there really *are* 4 chars in the field, one of them just isn't on line one, as far as livecode's is concerned. It's in some no man's land between line one and the non-existent line 2. I was saying that I think that is what led me to get confused about how filter should behave, even though I do understand how returns work.

I know that it's really hiding there, invisible to the first two questions, and that the right way to get the number of chars in the field is to use "the number of chars in field 1", rather than asking for the number of lines and the number of chars in each.. This is what I meant by getting different (all correct) answers based upon the scope of the question you ask.

My last post and the stack attached to it were a response to your asking me to count the chars, and to explain better what I was talking about.

[dunbarx]

Ah.

Now I see what you meant.

put "123" & return into fld "yourField"

answer numToChar(the last char of line 1 of fld "yourField") --gives "51" (the "3")
answer numToChar(the last char of fld "yourField") --gives "10" (the return)

And since there is only one line in the field, and line 2 is not only empty and nonexistent and has a length of 0, well, you already got the point.

Craig