Page 1 of 3
Params to Python external script?
Posted: Wed Jul 07, 2021 3:12 pm
by Zax
Hello,
As I have a long calculation to perform, I would like to try if a python script would be faster but I don't know how python and LC communicate.
Let's say i have the following LC script:
Code: Select all
on mouseUp
local tMinParam = 1
local tMaxParam = 1000
local tResult
repeat with x = tMinParam to tMaxParam
... do something and modify tResult
end repeat
answer tResult
end mouseUp
How can I pass the loop job to a python script with tMinParam, tMaxParam, and retreive tResult?
(in fact, I don't anything to Python, but a friend of mine can write Python script for me
)
Thank you.
Re: Params to Python external script?
Posted: Wed Jul 07, 2021 7:15 pm
by Bernard
Code: Select all
python -c "import sys; print 'hi',sys.argv[1], sys.argv[2], 'bye'" abc 'de f'
returns
hi abc de f bye
I'm sure your friend will want to pass the name of a .py script rather than -c switch.
So
Code: Select all
put shell("python myfile.py " && tMinParam && tMaxParam) into tResult
should work to pass your variables into the python script, and the result of shell() will be what the python script returns (assuming that the python executable is installed on a path automatically found by your command terminal).
This also assumes that your tMinParam and tMaxParam are not strings containing characters that will confuse the python interpreter.
Re: Params to Python external script?
Posted: Wed Jul 07, 2021 7:47 pm
by FourthWorld
If you can post the Python script it would allow us to translate it to see if it's actually faster.
Re: Params to Python external script?
Posted: Thu Jul 08, 2021 9:42 am
by Zax
Thank you.
My LC script is about large string (imageData). If the Python version is faster, I'll post the two versions.
Re: Params to Python external script?
Posted: Thu Jul 08, 2021 10:15 am
by Zax
Bernard wrote: ↑Wed Jul 07, 2021 7:15 pm
This also assumes that your tMinParam and tMaxParam are not strings containing characters that will confuse the python interpreter.
Oops, I missed that part... So I will not be able to send an imageData to the Python script
EDIT : so, I could I send to Python a file name? Maybe with some quotes?
EDIT2: strings don't seem to be a problem, maybe as long they don(t contain quote?
Code: Select all
put "hello" into p1
put "World" into p2
put shell("python " & pythonScripFile && p1 && p2) into tResult
put tResult
This works.
Re: Params to Python external script?
Posted: Thu Jul 08, 2021 2:28 pm
by Zax
I now realise that the inital idea wasn't good, as it's rather stupid to pass an imageData as parameters.
Thanks all for your replies.
Anyway, I've learned how to call Python and I will use it in other projects.
Re: Params to Python external script?
Posted: Thu Jul 08, 2021 10:18 pm
by FourthWorld
If you're in a position to post both scripts it would be very interesting to see the differences in structure and performance.
Re: Params to Python external script?
Posted: Fri Jul 09, 2021 12:01 am
by Bernard
Zax wrote: ↑Thu Jul 08, 2021 2:28 pm
I now realise that the inital idea wasn't good, as it's rather stupid to pass an imageData as parameters.
Thanks all for your replies.
Anyway, I've learned how to call Python and I will use it in other projects.
You could use some conversion from binary to a text format in livecode and pass that encoded imageData, then in python re-convert to binary data and process the image.
Or you could save the imagedata to a file, then pass that filename to the python script, where python would read the binary data from that named file.
Or, as Richard suggests, explain what the problem is you're trying to speed up by using python. Could be that Livecode has got a way of doing it that will be fast enough for your purposes.
Re: Params to Python external script?
Posted: Fri Jul 09, 2021 9:55 am
by Zax
My purpose is to turn an image to greyscale, and set transparency on bright grey values.
I suppose the best way to do that would be to use some graphic libs ane send shell commands, but I don't want to use external libs.
Bernard wrote: ↑Fri Jul 09, 2021 12:01 am
Or you could save the imagedata to a file, then pass that filename to the python script, where python would read the binary data from that named file.
Don't you think reading and writing large files will be longer?
For informations, this is my LC (9.04) code. Sorry, it's not very easy to read, as I tried to optimize it (and it's french commented
)
Code: Select all
local arrParams
on preProcess
if ("button" is in the target) then set hilite of the target to true
set cursor to watch
------- reset PixEnd (delete image pour full reset si l'alphaData est corrompu)
if (there is an image arrParams["destName"]) then delete image arrParams["destName"]
create image arrParams["destName"]
set threeD of image arrParams["destName"] to (the threeD of image arrParams["sourceName"])
set borderWidth of image arrParams["destName"] to (the borderWidth of image arrParams["sourceName"])
set showBorder of image arrParams["destName"] to (the showBorder of image arrParams["sourceName"])
------- visuel
set the rect of image arrParams["destName"] to (the rect of image arrParams["sourceName"])
set the topLeft of image arrParams["destName"] to \
(the left of image arrParams["sourceName"] & "," & the bottom of image arrParams["sourceName"] + 6)
-------
put the milliseconds into fld "Timer" -- /////
end preProcess
on getParams
put "Pix" into arrParams["sourceName"]
put "PixEnd" into arrParams["destName"]
put the hilite of btn "ChkInvert" into arrParams["invert"] -- négatif
get the thumbPos of scrollbar "SliderLighten" -- de 0 (aucun éclaircissement) à 10 (éclaircissement maxi)
put round(it * 200 / 10) into arrParams["brightness"] -- de 0 à 200
get the thumbPos of scrollbar "SliderContrast" -- de 0 (aucune modif) à 10 (contraste maxi)
put it / 10 into arrParams["contrast"] -- de 0 à 1
put 0.30 into arrParams["xRed"] -- augmenter pour éclaircir les rouges (avec total = 1)
put 0.59 into arrParams["xGreen"] -- augmenter pour éclaircir les verts (avec total = 1)
put 0.11 into arrParams["xBlue"] -- augmenter pour éclaircir les bleus (avec total = 1)
put the width of image arrParams["sourceName"] into arrParams["w"]
put the height of image arrParams["sourceName"] into arrParams["h"]
end getParams
on postProcess
------- visual effect
show image "PixEnd"
put (the milliseconds - fld "Timer") && "ms" into fld "Timer"
if ("button" is in the target) then set hilite of the target to false
end postProcess
on mouseUp
getParams
preProcess
turnToGrayTranspa arrParams["sourceName"], arrParams["destName"], arrParams["w"], arrParams["h"], arrParams["xRed"], arrParams["xGreen"], arrParams["xBlue"], arrParams["brightness"], arrParams["contrast"], arrParams["invert"]
postProcess
end mouseUp
on turnToGrayTranspa pixSourceName, pixDestName, imgWidth, imgHeight, xRed, xGreen, xBlue, brightness, fc, invertBWboolean
put "" into log
addToLog log, imgWidth && "x" && imgHeight && "pixels"
addToLog log, "r = " & xRed && " g = " & xGreen && " b = " & xBlue
addToLog log, "brightness = " & brightness
----------------------------------------------------- prépa : négatif
if (invertBWboolean) then
put 255 into grayCorrection
else put 0 into grayCorrection
addToLog log, "invertBWboolean = " & invertBWboolean && " --> grayCorrection = " & grayCorrection
----------------------------------------------------- prépa : contraste
put 0 - (fc^5 + 0.05) into fp5 -- pré-calcul
put sqrt(fc) into racf -- pré-calcul
addToLog log, "fc = " & fc
----------------------------------------------------- prépa : image et transparence
put the imageData of image pixSourceName into tImage
local aImage -- pour alphaData
-----------------------------------------------------
local tPixel, tImgPosition, tGray
# repeat for all the pixels in the image
repeat with tPixel = 0 to ((imgWidth * imgHeight) - 1)
----------------------------------------------------- extractions des composantes RGB
# multiply the image pixel position by 4 (Alpha + Red + Green + Blue)
put tPixel * 4 into tImgPosition
/* plus long
# calculate the gray level
put charToNum(char(tImgPosition + 2) of tImage) into tRed
put charToNum(char(tImgPosition + 3) of tImage) into tGreen
put charToNum(char(tImgPosition + 4) of tImage) into tBlue
----------------------------------------------------- gris
# here you can use any grayscale formula
# set the RGB of the pixel to the same value this gives us the gray level (0-255)
--put (xRed * tRed) + (xGreen * tGreen) + (xBlue * tBlue) into tGray
--------------------- application auto "négatif"
put abs(grayCorrection - ((xRed * tRed) + (xGreen * tGreen) + (xBlue * tBlue))) into tGray
*/
------------ calcul valeurs gris et application auto "négatif"
put abs(grayCorrection - ((xRed * charToNum(char(tImgPosition + 2) of tImage)) + \
(xGreen * charToNum(char(tImgPosition + 3) of tImage)) + \
(xBlue * charToNum(char(tImgPosition + 4) of tImage)))) into tGray
----------------------------------------------------- contraste
if (fc > 0) then
put floor(tGray * (1 - racf)) + racf * (255 / (1 + (0.9921875 * exp(fp5 * (tGray - 128))))) into tGray
-- Rislon : (x * (1 - sqrt(fc))) + sqrt(fc) * (255 / (1 + (0.9921875 * exp(-(fc^5 + 0.05) * (x - 128)))))
/* -- plus précis mais plus long
if (fc >= 0.5) then
put exp(-2 * (tGray - 128) * fc^4) into tmg
put floor(128 + 127.5 * (1 - tmg) / ( 1 + tmg)) into tGray
else
put exp(-2 * (tGray - 128) * 0.5^4) into tmg
get 128 + 127.5 * (1 - tmg) / ( 1 + tmg)
put floor(tGray * (1 - 2 * fc) + 2 * fc * it) into tGray
end if
*/
end if
----------------------------------------------------- éclaircissement général
put min(255, brightness + tGray) into tGray
----------------------------------------------------- application des nouvelles valeurs de gris
put numToChar(tGray) into ntcGray
put ntcGray & ntcGray & ntcGray into char (tImgPosition + 2) to (tImgPosition + 4) of tImage
----------------------------------------------------- transparence toutes valeurs (alphaData)
--------- transparence proportionnelle à la valeur de gris : plus c'est clair -> plus c'est transparent
/*
binaryEncode("C",0) --> transparentPixel
binaryEncode("C",255) --> opaquePixel
*/
put binaryEncode("C", 255 - tGray) after aImage -- 0 = transparent et 255 : opaque
end repeat
-------------------------------------------------------- affectation image dest
set the imageData of image pixDestName to tImage
set the alphaData of img pixDestName to aImage
if (there is a fld "Log") then put log into fld "Log" --/////
end turnToGrayTranspa
on addToLog @log, str
put str & return after log
end addToLog
function func0 x, f -- fcn Rislon pour f >= 0.5 : UNUSED telle quelle
return (128 + 127.5 * (1 - exp(-2 * (x - 128) * f^4)) / ( 1 + exp(-2 * (x - 128) * f^4)))
end func0
function func1 x, f -- fcn Rislon pour f < 0.5 : UNUSED telle quelle
return (x * (1 - 2 * f) + 2 * f * func0(x, 0.5))
end func1
Re: Params to Python external script?
Posted: Mon Jul 12, 2021 12:13 pm
by Bernard
Image processing isn't something I ever do. However, as no-one else has made any suggestions, I'll have a shot.
Can you use your logging to work out where this is slow? My suspicion is that it will be in your loop where you turnToGray by pixel.
If so then you may get an improvement by pulling out the 4 pixels on which you work and appending them to a new variable, rather than repeatedly addressing (groups of 4) pixels in the imagedata. Then set the imageData of your image to the data in the new variable.
Here's a high level example:
Code: Select all
on turnToGrayTranspa
put the imagedata of img 1 into t1
put zero into tPixCount
repeat for each char tPix in t2 -- only makes one pass through imagedata
add 1 to tPixCount
if tPixCount = 4 then
-- do pixel processing here
-- append processed pixels to t2
put zero into tPixCount
end if
end repeat
set the imageData of image 1 to t2 -- ty2now contains your new imageData
end turnToGrayTranspa
Those who do image processing may be able to come up with something better.
Re: Params to Python external script?
Posted: Mon Jul 12, 2021 5:45 pm
by Zax
Bernard wrote: ↑Mon Jul 12, 2021 12:13 pm
My suspicion is that it will be in your loop where you turnToGray by pixel.
You're right Bernard: a huge count of
charToNum and
numToChar take some time.
I tried the "after" method to fill an empty variable but found that in my case, process was longer.
Process is a little bit faster with a
repeat... step 4.
Re: Params to Python external script?
Posted: Mon Jul 12, 2021 6:15 pm
by Bernard
Can you give us an idea of how long things take?
Also I'm surprised that a single pass through the imageData takes longer than repeated scans of it looking for sets of pixels. Perhaps include your modified turnToGrayTranspa handler so others can see.
Another technique is memoization. Once you have done a calculation for 4 pixel values, you don't need to recalculate when you run into those 4 identical values again. You could use an array as a lookup table (the original values would be the lookup key, the calculated values the associated value of the array). Obviously this will give huge performance benefits on the image you included in your post, since it is bars of the same colour, but will offer less benefit with an image that has more random pixel data.
Finally, since your interest has changed from Python to LC performance, you might be better asking a new question specifically about your turnToGrayTranspa handler, and see if you can get the attention of someone who specialises in the area of image manipulation.
Re: Params to Python external script?
Posted: Mon Jul 12, 2021 7:32 pm
by FourthWorld
If I ask a fourth time might I have a copy of the Python script that was originally used? If it has some sort of licensing restriction I'd understand, but I would love the opportunity to make stylistic and performance comparisons of a non-trivial algo between the two languages.
Re: Params to Python external script?
Posted: Tue Jul 13, 2021 7:34 am
by Zax
Sorry
Richard if I wasn't clear (me english is rather poor) : I don't have Python script. A soon I realised passing an imageData as parameter was a bad idea, I told my fried not to write a Python script.
@
Bernard: process time depends of course of source image, and CPU.
For exemple, it takes 10 secondes for a 2500x600 pixels JPEG image with an old MacBook Pro.
Concerning memoization, the image shown on my screen capture was just an example to see gray values.
This is the actual code:
Code: Select all
local arrParams
on preProcess
if ("button" is in the target) then set hilite of the target to true
set cursor to watch
------- reset PixEnd (delete image pour full reset si l'alphaData est corrompu)
if (there is an image arrParams["destName"]) then delete image arrParams["destName"]
create image arrParams["destName"]
set threeD of image arrParams["destName"] to (the threeD of image arrParams["sourceName"])
set borderWidth of image arrParams["destName"] to (the borderWidth of image arrParams["sourceName"])
set showBorder of image arrParams["destName"] to (the showBorder of image arrParams["sourceName"])
------- visuel
set the rect of image arrParams["destName"] to (the rect of image arrParams["sourceName"])
set the topLeft of image arrParams["destName"] to \
(the left of image arrParams["sourceName"] & "," & the bottom of image arrParams["sourceName"] + 6)
-------
put the milliseconds into fld "Timer" -- /////
end preProcess
on getParams
put "Pix" into arrParams["sourceName"]
put "PixEnd" into arrParams["destName"]
put the hilite of btn "ChkInvert" into arrParams["invert"] -- négatif
get the thumbPos of scrollbar "SliderLighten" -- de 0 (aucun éclaircissement) à 10 (éclaircissement maxi)
put round(it * 200 / 10) into arrParams["brightness"] -- de 0 à 200
get the thumbPos of scrollbar "SliderContrast" -- de 0 (aucune modif) à 10 (contraste maxi)
put it / 10 into arrParams["contrast"] -- de 0 à 1
put 0.30 into arrParams["xRed"] -- augmenter pour éclaircir les rouges (avec total = 1)
put 0.59 into arrParams["xGreen"] -- augmenter pour éclaircir les verts (avec total = 1)
put 0.11 into arrParams["xBlue"] -- augmenter pour éclaircir les bleus (avec total = 1)
put the width of image arrParams["sourceName"] into arrParams["w"]
put the height of image arrParams["sourceName"] into arrParams["h"]
end getParams
on postProcess
------- visual effect
show image "PixEnd"
put (the milliseconds - fld "Timer") && "ms" into fld "Timer"
if ("button" is in the target) then set hilite of the target to false
end postProcess
on mouseUp
getParams
preProcess
turnToGrayTranspa arrParams["sourceName"], arrParams["destName"], arrParams["w"], arrParams["h"], arrParams["xRed"], arrParams["xGreen"], arrParams["xBlue"], arrParams["brightness"], arrParams["contrast"], arrParams["invert"]
postProcess
end mouseUp
on turnToGrayTranspa pixSourceName, pixDestName, imgWidth, imgHeight, xRed, xGreen, xBlue, brightness, fc, invertBWboolean
put "" into log
addToLog log, imgWidth && "x" && imgHeight && "pixels"
addToLog log, "r = " & xRed && " g = " & xGreen && " b = " & xBlue
addToLog log, "brightness = " & brightness
----------------------------------------------------- prépa : négatif
if (invertBWboolean) then
put 255 into grayCorrection
else put 0 into grayCorrection
addToLog log, "invertBWboolean = " & invertBWboolean && " --> grayCorrection = " & grayCorrection
----------------------------------------------------- prépa : contraste
put 0 - (fc^5 + 0.05) into fp5 -- pré-calcul
put sqrt(fc) into racf -- pré-calcul
addToLog log, "fc = " & fc
----------------------------------------------------- prépa : image et transparence
put the imageData of image pixSourceName into tImage
local aImage -- pour alphaData
-----------------------------------------------------
local tPixel, tImgPosition, tGray
# repeat for all the pixels in the image
repeat with tImgPosition = 0 to 4*((imgWidth * imgHeight) - 1) step 4
----------------------------------------------------- extractions des composantes RGB
# multiply the image pixel position by 4 (Alpha + Red + Green + Blue)
------------ calcul valeurs gris et application auto "négatif"
put abs(grayCorrection - ((xRed * charToNum(char(tImgPosition + 2) of tImage)) + \
(xGreen * charToNum(char(tImgPosition + 3) of tImage)) + \
(xBlue * charToNum(char(tImgPosition + 4) of tImage)))) into tGray
----------------------------------------------------- contraste
if (fc > 0) then
put (tGray * (1 - racf)) + racf * (255 / (1 + (0.9921875 * exp(fp5 * (tGray - 128))))) into tGray
end if
----------------------------------------------------- éclaircissement général
put min(255, brightness + tGray) into tGray
----------------------------------------------------- application des nouvelles valeurs de gris
put numToChar(tGray) into ntcGray
put ntcGray & ntcGray & ntcGray into char (tImgPosition + 2) to (tImgPosition + 4) of tImage
----------------------------------------------------- transparence toutes valeurs (alphaData)
--------- transparence proportionnelle à la valeur de gris : plus c'est clair -> plus c'est transparent
put binaryEncode("C", 255 - tGray) after aImage -- 0 = transparent et 255 : opaque
end repeat
-------------------------------------------------------- affectation image dest
set the imageData of image pixDestName to tImage
set the alphaData of img pixDestName to aImage
if (there is a fld "Log") then put log into fld "Log" --/////
end turnToGrayTranspa
on addToLog @log, str
put str & return after log
end addToLog
Re: Params to Python external script?
Posted: Tue Jul 13, 2021 10:48 am
by Bernard
I've just done a test and it took < 4 seconds to loop through an imagedata that had 4 million chars in it using "repeat for each". I wasn't doing all the various calculations you are doing, I was just running charToNum() on each char. I'm testing speed of loop format rather than anything connected to imaging.
Using your loop format was taking so long that I stopped after 4 seconds, and it had only processed 75,000 chars. This is the kind of difference one would expect.
Can you show us the times for "repeat for each" and show us the handler using that loop format. Contrast this with your loop.
Memoization might still be of use in your situation. Also I'd suggest you try speed comparisons using the kind of image that you think you will eventually process.