Page 1 of 1
Performance and background jobs
Posted: Thu Jan 31, 2019 11:45 am
by ace16vitamine
Dear all,
I have a simple (and stupid) question.
My application is like this one:
Code: Select all
on open_sock
Transfer file bla bla
sql_select
end open_sock
on sql_select
select bla bla bla
calculate something
show_message
end sql_select
on show_message
answer bla bla bla
end show_message
On "sql_select" I am calculating thinks, this takes about 20-40 secs. During this time the application GUI Frontend seems to be frozen, thinks are not calculating in background and there is enough CPU Power left. sql_select is integrated to the frozen card, I think this is the problem.
What can I do that operations like this are running in background?
Thanks
Stefan
Re: Performance and background jobs
Posted: Thu Jan 31, 2019 12:11 pm
by Klaus
Hi Stefan,
unfortunately LC is single-threaded, so there is not much you can do about this.
Maybe you can "cut" your long handler into smaller units?
Maybe you can post the handler and we check if we could make it faster and/or see
where we can cut it into little pieces.
Best
Klaus
Re: Performance and background jobs
Posted: Thu Jan 31, 2019 2:52 pm
by ace16vitamine
Hi Klaus (again),
this is some of the biggest block. I am merge some SQL outputs different times to create a laaaaarge variable.
I am starting with the Order IDs from t_array["vBestellnummer"] to get a list with all existing Order IDs. Then, in the first section, I am matching each line of s_array["vSendungsnummern"] and find the right ID which a matching to create a new variable (v_order).
The selects are always the same:
ID TAB TrackingID (or if paid, or name, surname etc)
I THINK the Problem is that it is very heavy to calculate this again and again... Maybe we can make this faster (and shorter).
Code: Select all
####### SECTION 1
put t_array["vBestellnummer"] into tNames
put s_array["vSendungsnummern"] into tIds
#Example t_array["vBestellnummer"]
#AU123
#AU456
#AU789
#Example s_array["vSendungsnummern"]
# AU123 A123
# AU456 B456
set itemdel to TAB
## Loop through all NAME pairs
repeat for each line tLine in tNames
put item 1 of tLine & TAB into tItem
put lineoffset(tItem,tIds) into tOffset
## No match, modify this line if you want to add an empty ITEM
## to the current NAME line or whatever you want to do in that case
if tOffset = 0 then
put tab into item 2 of line tOffset of tIds
put tLine & TAB & item 2 of line tOffset of tIds & CR after v_order
next repeat
else
## repeat for each is READ only, so we just create a new list form the resulting data:
put tLine & TAB & item 2 of line tOffset of tIds & CR after v_order # [b]<- This is the Target for section 1[/b]
end if
end repeat
delete char -1 of v_order
######### RESULT:
# AU123 A123
# AU456 B456
######################################## Next Section: Match if DHL, DPD...
####### SECTION 2
#Example t_array["vVersandart2"]
#AU123 DHL
#AU456 DPD
#AU789 GLS
#Example v_order (result from the 1st section)
# AU123 DHL123
# AU456 DHL456
put v_order into tNames
put t_array["vVersandart2"] into tIds
set itemdel to TAB
## Loop through all NAME pairs
repeat for each line tLine in tNames
put item 1 of tLine & TAB into tItem
put lineoffset(tItem,tIds) into tOffset
## No match, modify this line if you want to add an empty ITEM
## to the current NAME line or whatever you want to do in that case
if tOffset = 0 then
next repeat
else
## repeat for each is READ only, so we just create a new list form the resulting data:
put tLine & TAB & item 2 of line tOffset of tIds & CR after v_order2 # [b]<- This is the Target for section 2[/b]
end if
end repeat
delete char -1 of v_order2
######### RESULT:
# AU123 A123 DHL
# AU456 B456 DPD
######################################## Next Section: Match if paid
####### SECTION 3
#Example t_array["paid"]
#AU123 yes
#AU456 no
#AU789 no
#Example v_order2 (result from the 2st section)
# AU123 DHL123 A123
# AU456 DHL456 B456
# (..) with v_order3, v_order4, v_order5, v_order6
Re: Performance and background jobs
Posted: Thu Jan 31, 2019 5:18 pm
by Klaus
Hm, OK, I would probably use some local variables or custom properties
and split the long handler into 6 smaller handlers -> v_order1 to v_order6.
And use SEND to call them from the previous handler, maybe with a little delay
This way you can shorten the processing time, but may still take a couple
of seconds for every hanbdler, so I would show the user a "fake dialog"
(a group or whatever) indicating that the app is not dead but working.
Seeing is believing!
Know what I mean?
Re: Performance and background jobs
Posted: Thu Jan 31, 2019 5:50 pm
by AndyP
I its a long process then this could be done in a separate hidden stack launched from the main client facing stack. You could then use socket server and client to communicate between the stacks and pass the results back to the main stack. Examples of client and server stacks can be found here
http://livecodeshare.runrev.com/stack/5 ... ket-server
http://livecodeshare.runrev.com/stack/5 ... ket-client
and lessons
http://lessons.livecode.com/m/4071/l/12 ... ng-sockets
Re: Performance and background jobs
Posted: Thu Jan 31, 2019 5:53 pm
by Klaus
Hi Andy,
hm, great idea, but the ENGINE is single threaded, so the same engine running two stacks will cause the same problems as using only one stack, not?
Best
Klaus
Re: Performance and background jobs
Posted: Thu Jan 31, 2019 8:58 pm
by ace16vitamine
Klaus wrote: ↑Thu Jan 31, 2019 5:18 pm
Hm, OK, I would probably use some local variables or custom properties
and split the long handler into 6 smaller handlers -> v_order1 to v_order6.
And use SEND to call them from the previous handler, maybe with a little delay
OK, i do this later (in the case that my wife is sleeping... )
This way you can shorten the processing time, but may still take a couple
of seconds for every hanbdler, so I would show the user a "fake dialog"
(a group or whatever) indicating that the app is not dead but working.
Seeing is believing!
Know what I mean?
Yes, BUT... Ive already installed a spinner. In the case that my "construction" is working the spinner hangs
OK, first, let me check if this is really my problem and not a slow DB Connection. I can put whatever to the message box to check on which step the construction is.
Re: Performance and background jobs
Posted: Thu Jan 31, 2019 9:14 pm
by Klaus
Yes, BUT... Ive already installed a spinner. In the case that my "construction" is working the spinner hangs.
Well, a repeat loop in general takes all CPU power, UNLESS you add a little "wait" statement:
Code: Select all
...
repeat for each line tLine in tNames
## Do your stuff here...
## ...
wait 0 with messages
end repeat
...
That should give the SPINNER time to do its job, at least worth a try.

Re: Performance and background jobs
Posted: Thu Jan 31, 2019 10:36 pm
by ace16vitamine
Well, a repeat loop in general takes all CPU power, UNLESS you add a little "wait" statement:
Hm... good to know!
But I have another idea to get more performance
I have a key... The Order Number. The key is always the same. I can put the SQL Select in another "Transfer" SQLite DB. That means instead of the beast the CPU Power will take over to SQL.
But I need to do this for each line... :-/. I am not sure if the SQLlite Connection is really faster (maybe, a little bit) because in this case (for each line) I have hundreds of inserts
Let me first check to split it into different handlers and then call the handler from each before.
Re: Performance and background jobs
Posted: Thu Jan 31, 2019 10:50 pm
by FourthWorld
Mutli-threading isn't an option, but multi-processing is. We could explore that if needed, but I'm wondering if the query itself could be optimized. 40 seconds seems like a long time. Can you tell us more about the query, and the number of records, and whether the query involved unindexed fields?
Re: Performance and background jobs
Posted: Fri Feb 01, 2019 6:39 am
by AndyP
Klaus wrote: ↑Thu Jan 31, 2019 5:53 pm
Hi Andy,
hm, great idea, but the ENGINE is single threaded, so the same engine running two stacks will cause the same problems as using only one stack, not?
Best
Klaus
Yes, but once compiled, each instance will be its own entity and allocated its own slice of processor and memory.
Re: Performance and background jobs
Posted: Fri Feb 01, 2019 7:00 am
by AndyP
But I need to do this for each line... :-/. I am not sure if the SQLlite Connection is really faster (maybe, a little bit) because in this case (for each line) I have hundreds of inserts
Another option might be to offload these inserts to a server script and let the server do the heavy lifting .
Re: Performance and background jobs
Posted: Fri Feb 01, 2019 7:05 am
by AndyP
Ah..just noticed its a sqlite db and so probably local..in which case ignore my last post.
Re: Performance and background jobs
Posted: Fri Feb 01, 2019 3:21 pm
by AxWald
Hi,
ace16vitamine wrote: ↑Thu Jan 31, 2019 10:36 pm
[...] hundreds of inserts [...]
Oh oh. You will want to avoid this, it's a crazy slowdown! Better CREATE TEMPORARY TABLE and feed it the data all at once, then move it around inside the db (UPDATE, INSERT).
Anyways, I don't really understand what you're doin here, but to me it looks like it should be done within the database itself. This is almost always the better way - databases are optimized to do such things. You might need to make yourself familiar with things like JOIN, UNION, temporary tables, stored procedures & views - but once done you'll not want to miss it anymore.
Hmm. I read your example again. Pity you left out the SQL so I cannot visualize the data. But I can't help thinking you could replace all this with a single SELECT ... FROM ... JOIN statement?
Have fun!
Re: Performance and background jobs
Posted: Sun Feb 03, 2019 8:09 pm
by ace16vitamine
Hey guys,
thank you for all your informations.
Ive checked my App step by step to identify WHAT is the performance hungry monster.
What I found out... My "construct" needs < 1sec, the calculation is not the problem. The SQL Selects are also fast, the bottle neck is:
- The Hard Disc (Performance on virtual environment is extrem slow, don't know why (tested on parallels and ESXi))
- The Sock opening process is slow and raised the CPU (open socket to sock_server with message "push_transferfile_socket_auth" takes 5-10 secs), the communication and transfer itself is fast (< 1 sec).
Now I need to check why the sock opening process is slow. There is no firewall or filter between both applications (localhost connection) and the problem comes only while I am open a socket. After the socket is open the system is running fast. If I open a connection with telnet the answer is immediately there.
Stefan