Loading Data into an array similiar to the Data Statement in BASIC

[diobosco]

Being an observer of this forum and everything related to Livecode since many years on below radar mode (incl. being a kickstart-supporter of the open-source initiative) I now found the courage to contribute to the forum. But being a new contributor to the forum it is part of the forum etiquette to say hello to the forum.

Hello to the forum!

While working on a project I needed to preload a bunch of values to a stack at initialisation phase at run-time. I couldn't get my head around throwing dozens and dozens of "put ... to tvar commands" in the initial section of the stack. From very early days I could recall that in programming languages like BASIC there was a DATA construct available (not that I am fond of BASIC but this feature was rather cool in some instances). However, this was excactly what I needed but I couldn't find a similiar language construct in LC.

Of course you could argue that I could put the values in an external file in XML or JSON or even flat text form. But I didn't want to have that because I am fond of the notion of a self-contained application and I try to avoid this whenever possible.

Skimming through the different LC forum posts I finally could find a threat of someone who had exactly the same requirement. If you are interested you can find the thread under the subject: "Storing large amounts of data in a script only stack".

In that thread I became aware for the very first time that with LC it is possible to consumate the very own script and do some neat processing with it. While potentially a dangerous thing to do it offered a way for me to solve my requirement pumping a bunch of predefined "hard-coded" values into an array.

Based on that idea I developed a script that reads data lines in CSV notation where the data are encapsulated in the LC comments. But I didn't like three things about the proposed solution in the thread mentioned:

• The absolute reference of the data lines. What happens when I reorganise the code and "line -5 to -1 of the script of me" doesn't contain the data lines involved any longer?

• I have different kind of types of data that needed to be loaded into different arrays

• The way of storing the data lines in the array seemed too limited for my purpose. I wanted to create different array index scheme.

So, I decided to create something new. As an excercise session I developed a generic script that enables the initial load of "fix" data in an array with 4 different array building schemes to address the following:

The suboptimal absolute referencing of data lines:
Instead of using an absolute adressing mechanism I defined that a data section needs to have a keyword (called DATASECTION) at the beginning of the data section block in the script and another one (with DATASECTION) at the end of the relevant data section block. Only data lines in between are becoming candidates acting as input for the array load.

Different kinds of different data types for different arrays:
I enhanced the DATASECTION block start and end line with an ID. Having a block start it begins one data section with:

Code: Select all

/* bla bla DATASECTION ID:DBCONFIG bla bla */
--DATA the data line 1 in the DB-config section
--DATA the data line 2 in the DB-config section
::
--DATA the last data line in the DB-config section
/* bla bla end of DATASECTION ID:DBCONFIG bla bla */


/* bla bla DATASECTION ID:NETWORKCONFIG bla bla */
--DATA the data line 1 in the network config section
--DATA the data line 2 in the network config section
::
--DATA the last data line in the network config section
/* bla bla end of DATASECTION ID: NETWORKCONFIG bla bla */

Please be aware of the ID next to ID: This enables the definiton of several, different data sections and it's loading into different arrays.

Implementing different ways of array and key structures
I had different requirements for array indexing.

• an array with 1D key dimension using item 1 as it's key and the rest of the data line being in ONE field

• an array with 2D key dimension using running numbers (starting with 1) as it's key for both dimensions

• an array with 2D key dimension using item 1 as it's 1st-order-key and a running number (starting with 1) as it's 2nd-order-key

• an array with 2D key dimension where item1 of each data line acts as 1st-order-key and the column / item acts as the 2nd-order-key (much like a spreadsheet addressing scheme (A1, or C7)

So I implemented 4 different modes of array loading:

• STANDARD

• NUMERIC

• KEYVALUE

• MATRIXRC

The way to communicate to the script in which of the 4 array types the data needs to be loaded I created the DATAFORM statement. It's format is:

Code: Select all

--DATAFORM (STANDARD|NUMERIC|KEYVALUE|MATRIXRC)

The DATAFORM statement is optional. When the DATAFORM statement is missing STANDARD is assumed (default). THE DATAFORM statement can be placed anyway between the 2 DATASECTION lines. When more than one DATAFORM statements are specified the first appearance will be taken.

So, this is a well-formed data section entry:

Code: Select all

/* ------ DATASECTION ID:DBCONFIG ------- */
--DATAFORM KEYVALUE
--DATA item1,item2,....,itemN
--DATA item1,item2,....,itemN
::
--DATA item1,item2,....,itemN
/* ------ END of DATASECTION ID:DBCONFIG ------- */

First, in order to enable the function you need to put the script as substack of your main stack and put the "start using LoadData2Array" in the preopenstack handler.

Second, locate the stackname where you have your data section. The stackname needs to be placed as parameter 2 of the function call. This parameter is optional.

Third, call the function LoadData2Array with the DATAFORM name of the DATASECTION as parameter 1 and the stackname (where the data section is located) as the second parameter. When parameter 2 isn't provided LoadData2Array looks for the data section in it's own script.

The call of the function is as follows (example):

Code: Select all

put LoadData2Array("LD2ASTAND", "Load Data 2 Array Demo") into StandardArray.

Next to the library script I've created a demo stack where you can find examples of the different data sections and the different calls of LoadData2Array. The function itself is a substack of the demo stack. When testing you should create a breakpoint before terminating the handler and inspect the array created.

I hope this script is of value for you.

Diobosco

[bogs]

Hello Diobosco, welcome to the forum (finally)

The method you arrived at is interesting, but seems a hard way to go at it. After all, you could put the hard-coded data into one (or several) custom properties, and then either use it directly or put each into an array using combine.

If the data changes, and you are using the stack and launcher method, you could use split on the array to update the custom properties.

Of course, so simple a suggestion might run afoul of your 3rd point :
* The way of storing the data lines in the array seemed too limited for my purpose. I wanted to create different array index scheme.

...but I haven't read the referenced thread yet, so it may not.

Like I said earlier, though, it is an interesting look at it.

[dunbarx]

Hi.

Can you give a short example, including just a little bit of data?

I am with Bogs, that there are simpler methods. But I would need a test case to get a better idea of what you are up to.

Craig

[diobosco]

@bogs: I am aware of custom properties which I really don't like. You won't believe it but I started with them in my project and discarded them after the handling of them in the IDE drove me nuts. I am not talking about dealing with them on a script level. That is easy but managing them in the IDE brought me to insanity.

Being more specific, the reason might be that I am not proficient enough to use them in my way of developing an application but I don't like them because of 2 reasons:

• Entering the keys and values of a custom property using the IDE is - at least for me - a pain in the extended back. Entering them is very clumsy - at least to my taste. Entering one entry takes some time, then after entering 10 values I am like "aagh, I forgot one after position 5. How can I position the forgotten one after pos 5? Doesn't seem to be possible. OK, then delete the custom properties from pos 6 to 10 just to enter the forgotten one. So I couldn't find out how to insert one, I was only able to append it in the list.

• But my major critic point is that I cannot see them during development. My way of devloping is that I use the project browser a lot. There I have all elements in one window: The stack(s) and substacks, the cards and inside the cards all objects like buttons, etc. Next to each object in the project window I can see whether it has code associated to it (indicated by the number right to the objects name). Now, where can I see whether any object has "variables" in the form of custom properties? Maybe I am not proficient enough and it shows it somewhere but I couldn't find them. While this is not a huge problem during development ("obviously I know my stuff, right?" ) it becomes messy when I get back to the developed application one or two years later. Or even worse how can I find out about all the custom properties that are referenced in the stack which was developed by someone else? Opening up one object after another in the property inspector and clicking the second or third button?

Again, it might be that "I did beat the wrong problem", just because I don't know how deal with them effciently but at least this is the rational behind my desperate approach to pump the initial values in the application.

@dunbarx: Sure.
Currently, I am sitting on an expression evaluator (you have a mathematical expression in a text field and my stack interprets the term and perform the calculation). BTW: I am aware of the expression evaluator of Andreas Rozek and I am also aware that I could do it with the DO command but I want to learn the fundamentals of the algorithm so I am developing my own. In order to do so, I need a couple of tables with the "rules of mathematics". One of them is the precedence of operators. Eg. having the term "1+2*3 the operator precedence rule is telling that first it needs to conduct the multiplication and then the addition because * has a higher priority than a +. For that I have defined a data table which looks like (not final, WIP):

Code: Select all

/*-------------------DATASECTION ID:OPERATORPRECEDENCE-------------------
Specifies the operator precedence, the number of operands and its associativity
DATA operatorsymbol,prority,numoperands,associativity
-------------------------*/
--DATAFORM STANDARD
--DATA ^,40,2,right
--DATA /,30,2,left
--DATA %,30,2,left
--DATA *,30,2,left
--DATA +,20,2,left
--DATA -,20,2,left
--DATA <,10,2,left
--DATA >,10,2,left
--DATA >=,10,2,left
--DATA <=,10,2,left
--DATA !=,7,2,left
--DATA <>,7,2,left
--DATA =,7,2,left
--DATA &&,4,2,left
--DATA ||,2,2,left
/*----------------END DATASECTION ID:OPERATORPRECEDENCE-------------*/

Item 1 the operator
Item 2 the priority of the operator
Item 3 the number of operand for this operator (probably not necessary)
Item 4 the associativity of the operator

The key to this 1D table will be item1 (meaning the operator symbol).

A more complex example is a 2D table that holds a cross matrix with operators:

Code: Select all

/*-------------------DATASECTION ID:OPERATORCROSS-------------------
Specifies the possibilites for two operators directly after another.
Example would be Operator * followed by a - is allowed (example:2*-4), A "*" directly after "*" is not allowed.
At the vertical axis all possible operators at the first position are listed
At the horizontal axis all possible operators at the second position are listed
A 1 at the intersection of a specific row and column indicates that this combination
is allowed. All other entries indicates that this combination isn't allowed.
--------------------------------*/
--DATAFORM MATRIX
--DATA     ,-,+,*,/,%,(,),FUNC
--DATA    -,1,1,-,-,-,1,-,1
--DATA    +,1,1,-,-,-,1,-,1
--DATA    *,1,1,-,-,-,1,-,1
--DATA    /,1,1,-,-,-,1,-,1
--DATA    %,1,1,-,-,-,1,-,1
--DATA    (,1,1,-,-,-,1,1,1
--DATA    ),1,1,1,1,1,1,1,-
--DATA FUNC,-,-,-,-,-,1,-,-
/*-----------------END DATASECTION ID:OPERATORCROSS----------------*/

Item1 is 1st-order key, the items on line 1 are the 2nd-order key. Example:

Code: Select all

put "*" into tOperator1
put "-" into tOperator2
if operatorArrayA[tOperator1][tOperator2] = "1" then
   -- is allowed
else
   -- not allowed.
end if

I hope this clarifies the use case for the tables.

[diobosco]

Hello Diobosco, welcome to the forum (finally)

The method you arrived at is interesting, but seems a hard way to go at it. After all, you could put the hard-coded data into one (or several) custom properties, and then either use it directly or put each into an array using combine.

If the data changes, and you are using the stack and launcher method, you could use split on the array to update the custom properties.

Of course, so simple a suggestion might run afoul of your 3rd point :
* The way of storing the data lines in the array seemed too limited for my purpose. I wanted to create different array index scheme.

...but I haven't read the referenced thread yet, so it may not.

Like I said earlier, though, it is an interesting look at it.

Sorry, I answered to your post in a separate message in this theread. Pls see there.

[diobosco]

Hi.

Can you give a short example, including just a little bit of data?

I am with Bogs, that there are simpler methods. But I would need a test case to get a better idea of what you are up to.

Craig

Hi Craig
Sorry, I answered to your post in a separate message in this thread. Pls see there.