What could be wrong in arrayDecode(base64Decode(tTempArray))?

[trevix]

My standalone is saving a user array log using

Code: Select all

put base64Encode(arrayEncode(gTempArray)) into tArray
put tArray into URL ("binfile:" & sLogPath) --save to file

The standalone uses "on errorDialog pExecutionError" in order to collect execution errors and send them to my cloud DB

Yesterday I received an error message from a user on line 2 of this

Code: Select all

 put URL ("binfile:" & sLogPath) into tTempArray--load the pref file
put arrayDecode(base64Decode(tTempArray)) into gTempArray --holds all the data to be updated locally

Clearly there was some special character in the array that broke one of the two commands (base64Decode or arrayDecode)
I don't think it is possible that the tTempArray wasn't an array.

While I cannot know the content of tTempArray, since was saved locally and, beside, the user since then has gone over it, is there a way so that, on saving, I can be sure that it can be read without error? (beside using a Try,end Try control)?

Thanks

[jacque]

Funny, I had a user with the same problem yesterday. I'm using the same method as you are, arrayEncode and then base64Encode. I don't have an answer for you but I'm logging all server interactions and when I looked at the base64 string it was missing the equals signs at the end. It wouldn't decode correctly without those.

We have hundreds of users and the issue has only happened once, so for now we're thinking it's a transmission error over the net.

My script does use a try/catch clause though. You can't just check for missing equal signs at the end because they're just filler and may or may not be required in a valid string.

[trevix]

Hello.
In my case cannot be the net since everything was done locally, saving and reading to a file.
In my case also, this error happened only one time over hundred of users and almost 2 years.
I am afraid it is some hidden LC bug I can't do anything about and impossibile to reproduce. Sort of like the fact that a script sometimes forgets which is the substack it is running on.
Being an English/Italian standalone, my first idea was about some special char (we use plenty of them in Italian) disliked by one of the two commands.
I wonder if they have been tested with all the possible ASCII combination. I doubt.

Oh well. Will do a try/catch.

[FourthWorld]

Being an English/Italian standalone, my first idea was about some special char (we use plenty of them in Italian) disliked by one of the two commands.

Check the full path to the file. Some older versions of LC choked on special chars in file names.

[jacque]

Tracing through my handler, it looks like base64Encode is the culprit or maybe base64Decode. I only have access to the base64 string in the log, so I'm not sure whether the problem happened during encoding or decoding. But the handler fails when trying to decode.

We don't have any special characters, it's mostly just numbers with a short 1 or 2 character prefix (v, ch, or b.)

Edit: BTW, which version of LC built the standalone? I'm using 9.6.5.

[trevix]

I am using LC 9.6.6 rc1

[jacque]

I just happened to think that arrayEncode inserts a lot of special characters, mostly unprintable ones. Maybe that's the problem. But only once in a while.

Do you have a copy of the original text? I was able to extract that from my debugging log. I wonder if a report is in order.

[trevix]

No I don't.
I was sent only the error report

ORIGINAL ERROR:
465,68,1
253,68,1
253,68,1
241,17,1,libraryStack
353,0,0,stack "lib_analytics" of stack "/data/app/com.trevix.it.segnapunto-zCwjNxcHSJ0YZF9KNwmCpw==/base.apk/mainstack"

As usual, beside line 68, the rest doesn't say that much

[FourthWorld]

No I don't.
I was sent only the error report

ORIGINAL ERROR:
465,68,1
253,68,1
253,68,1
241,17,1,libraryStack
353,0,0,stack "lib_analytics" of stack "/data/app/com.trevix.it.segnapunto-zCwjNxcHSJ0YZF9KNwmCpw==/base.apk/mainstack"

As usual, beside line 68, the rest doesn't say that much

There should be nothing usual about that. That error format is for machine parsing, and should never be user-facing. How did the user encounter that info?

Here's what the error codes mean:
465: put: error in expression
253: if-then: error in statement
253: if-then: error in statement
241: Handler: error in statement
353: Object Name:

Without seeing the script to provide context we can't do much more, but hopefully that will help.

Also: why use base64 there? It's provided for compliance with certain MIME specifications, but you can read and write LSON (encoded array) files just like any other binary file type.

[jacque]

That error format is for machine parsing, and should never be user-facing. How did the user encounter that info?

Trevix has an errorDialog handler that reports it. You can set that up in standalone settings so users can email errors to the developer. I do that routinely, it's very helpful.

Also: why use base64 there? It's provided for compliance with certain MIME specifications, but you can read and write LSON (encoded array) files just like any other binary file type.

In this case, simply saving an arrayEncoded string to disk would work fine. In my case, since I'm sending data over the net, I follow Andre Garzia's method:

arrayEncode
compress
base64Encode

I could probably skip the compression but Andre is usually right about such things so I adopted it.

[FourthWorld]

That error format is for machine parsing, and should never be user-facing. How did the user encounter that info?

Trevix has an errorDialog handler that reports it. You can set that up in standalone settings so users can email errors to the developer. I do that routinely, it's very helpful.

Trevix described it as not very helpful at all. Unless one has special tooling like the Error Lookup utility stack) it's effectively just meaningless gibberish.

It could become useful if the supplied dialog took an extra two lines of code to look up the error and translate it into something useful for humans to read, or provided a tool for devs to use when receiving reports from the field.

But if this is what the supplied dialog does (can't say; I've been using my own error handling for many years) it's sorely in need of some UX awareness.

Also: why use base64 there? It's provided for compliance with certain MIME specifications, but you can read and write LSON (encoded array) files just like any other binary file type.

In this case, simply saving an arrayEncoded string to disk would work fine. In my case, since I'm sending data over the net, I follow Andre Garzia's method:

arrayEncode
compress
base64Encode

I could probably skip the compression but Andre is usually right about such things so I adopted it.

I'd leave the compression. Gzip reduces at an average of about 30%, and Base64 adds about 30%, and with network transfer time playing a key role in an app's apparent performance at least that's a wash.

But knowing Andre, I'll bet he has a good reason for using that in the context where he did. It may be illuminating to know that context.

The author of the Base64 RFC describes its role as:

Base encoding of data is used in many situations to store or transfer data in environments that, perhaps for legacy reasons, are restricted to US-ASCII [1] data. Base encoding can also be used in new applications that do not have legacy restrictions, simply because it makes it possible to manipulate objects with text editors.

IIRC very old versions of LC (possibly as old as MC?) didn't have a means of setting HTTP headers. So without a way to specify a Content-Type of something like "application/x-binary", Base64 was a good way to work around that limitation. And back in the MC days HTTP 1.1 was just emerging (even the example mchttpd stack was HTTP 1.0, missing the Content-Type and Content-Length header fields, merely useful for text but essential for binary data), so the modest additional overhead of Base64 was a small price to pay in a world with few alternatives.

Whether one would still need Base64 would of course depend on what's on the receiving end. If the dev has no control over the receiver, Base64 will allow integration with even legacy systems well enough. But when the dev is making the receiver, omitting the Base64encode will save about a third of the transmission time, much more if compress is also used.

Either way, Trevix can safely drop Base64 with local files, saving both a little disk space and perhaps a lot of waiting for the bug fix. Do you know offhand if this Base64 bug has been reported yet?

[jacque]

Trevix described it as not very helpful at all. Unless one has special tooling like the Error Lookup utility stack) it's effectively just meaningless gibberish.

It wasn't useful because he didn't know the cause, but it was useful because he knew exactly where in the code it occurred. THe information allowed me to recognize the problem as similar to mine. The error list isn't meant for the user, who wouldn't be any more informed if they knew the error happened on line 68 in an if/then clause.

I don't know if Trevix uses it, but the way the built-in error reporting works is that the developer provides a user-friendly comment ("An error occurred, please let us know what you were doing when it happened" -- or something like that) and then, at the developer's discretion, offers to open an email or save a file to disk, or both, with the included error list. I generally choose email because users know how to send email but not all of them know how to send a file. The email client opens automatically with the developer's comment at the top, I leave a few spaces for the user to type their comments, and the error codes are supplied beneath that. It allows the developer to know what actions caused the problem (if the user is facile enough to explain it,) where in the code the error was, and if we're lucky, how to reproduce it.

Users don't care what the gibberish is or what it means, and mostly they don't look at it. They sometimes just cancel out and we never know anything went wrong.(Well, there was one 1-star review where the user said the app crashed constantly but never told us.) But in most cases they send the email to the automatically assigned email address and we know where to look and hopefully what to do. I've fixed dozens of obscure bugs armed with this info over the years.

I'd leave the compression. Gzip reduces at an average of about 30%, and Base64 adds about 30%, and with network transfer time playing a key role in an app's apparent performance at least that's a wash.

You're probably right, though it's never been much of an issue. So far I'm not sending large amounts of data, my current app sends 1-2K, if that. The one that errored recently was 18 bytes.

Do you know offhand if this Base64 bug has been reported yet?

I doubt it, and since we can't reproduce it, it would be very difficult to track down. I've been using this method in different apps for years and the problem happened only once the other day. The current app has been in use for a year and has hundreds of users without any issues. Because of the log I keep, I do have the user's returned server data. The user's data was intact for three logins (with the same data each time) and on the fourth and sequential instances became corrupted. I was able to decode the first three to see the original text, and after that decoding failed immediately after base64decode. I noticed the base64 string was different in the last ones.

That's why we thought it must be a network error. The same data from the same user was fine over three server interactions, and then wasn't. But now that Trevix has had the problem on a local device I'm not sure why it happened.

[trevix]

Hello.
Let me explain how my Standalone (mobile only) reporting works:

1 - The app collect user analytics in a global array and save them locally using

Code: Select all

command SaveTheArray
     put base64Encode(arrayEncode(gTempArray)) into tArray
     put tArray into URL ("binfile:" & sLogPath) --save to file
end SaveTheArray

sLogPath is something like specialFolderPath(tFolder) & "/SegnaPunto/_MicAn.log" where, on mobile, tFolder is "cache"

2 - the gTempArray analytics are sent to my Filemaker server (using its API) creating a new recod at each restart of the App. The data are those of the previous session. This is done this way since on mobile it is almost impossible to intercept when the App gets closed and, often enough, the devices are not online (for several reasons)

3 - the reason of the base64Encode choice was to save to a local file that was not easily messed up by the user and that could be transmitted from the App at a user request (in the App there is a card to allow the user to email the last user data).
I could probably just do a textEncode, but after many published version, it would be hard to change to a new method.
Note that data to FM server are only Jsonized and textencoded.

4 - the gTempArray contains info on:
-how log the user stay on each card
-what kind of device is running on
-version of app
-etc.

5 - When the App intercepts en error (using "on errorDialog") it advices the user if he is willing to share the error and, if yes, it put the LC error detail in the gTempArray. The user see only a "an error occurred". The developer receives all the data in the FM server record.

6 - when there is an error reported on my FM server, I can see where it happened and, using the "LC error lookup" plugin I can try to understand what went wrong.

7 - the reading of the gTempArray, happens on startup using:

Code: Select all

put URL ("binfile:" & sLogPath) into tTempArray--load the pref file
put arrayDecode(base64Decode(tTempArray)) into gTempArray --holds all the data to be updated local save

the second line is the one when the error of this post has been catch (no, I haven't reported it since I cannot reproduce it)
After the reading, the gTempArray is sent to my FM server, as I said Jsonized and textencoded.
Then the gTempArray is emptied and the App start again collecting data and writin them to gTempArray

8 - On "errorDialog" I cannot collect all the values of global and local vars,. Mostly because the App spans on multiple substacks and cards and it would be too much time consuming. In this case I could not know that I had to collect and send the same value of gTempArray that generated the error.

I did a test stack (see attachment) that approximately reproduce the write and read of all the special chars I could think off. It works, so I assume that special chars on encoding is not the culprit.

[FourthWorld]

Trevix described it as not very helpful at all. Unless one has special tooling like the Error Lookup utility stack) it's effectively just meaningless gibberish.

The error list isn't meant for the user, who wouldn't be any more informed if they knew the error happened on line 68 in an if/then clause.

Of course, but developers are people too.

In Trivix's case he's written his own custom error handling, so he knows his way around the error format.

But you and I see discussions every few months for years about mystifying error data, which is why we dug into the IDE to find what was needed to make the error lookup utility you use to parse such things.

Here's a couple examples of how the IDE presents raw error info:
viewtopic.php?f=19&t=36553&hilit=error+dialog#p211369
https://www.mail-archive.com/use-liveco ... 11059.html

I write my own error handling, and you use a custom lookup tool. So this sort of thing isn't an issue for you or me.

But how many thousands of LC devs don't use the lookup tool. How many thousands more don't even know it exists?

Is it asking too much for a one-liner lookup to be added to human-facing error reporting? ExecutionContexts would help too, even better with tabular format with headings for easy reading. Top to bottom about two hours' work max.

Bonus points if they want to also provide static analysis tools that can take that info and not only do the lookup, but also take the dev to the error line in the script and even do some static analysis to provide hints for a fix.

But I'm not expecting Clang-level error handling in LC. It would be super, and Swift and others have depth like that, but I'm aiming lower than those competing tools.

I'm just advocating for the dozens of users you and I keep seeing who are mystified when the LC IDE presents answer dialogs dumping machine-oriented raw error info with no further assistance.

If I'm aiming for a high user experience, guilty as charged. I'd really like this tool to be as popular as it can be; change in customer acquisition rates will come with change in the customer experience.

[FourthWorld]

Hello.
Let me explain how my Standalone (mobile only) reporting works:
...
2 - the gTempArray analytics are sent to my Filemaker server (using its API)

Thanks for that background, Trivix. Agreed, with older tech like Filemaker it's probably useful to have the base64 there.

Too bad, though, if there's a bug in LC's base64encode. I had hoped we might have a way to bypass it.

Have you arrived as a sample set of data known to fail with that function?