How can an image be searched?

[ajperks]

A page of text can easily be searched to find to position of a specific character. However, I need to search an arbitrary image, let's say the screen or a JPG for the specific occurrence of a coloured pixel or mousepointer or simple shape and report it's x,y position.
Has anyone done this?
If so, how?

Reason, I am exploring the possibility of following the path of a simple robot captured by webcam.
In robotics at toy or vacuum cleaner level, accurate indoor positioning is complex and expensive, resorting to lidar and dual cameras along with the necessary code. FYI, using an accelerometer for direction and wheel rotation for distance quickly becomes inaccurate due to slippage.

In the simplest terms, for my purposes, to know where the robot is in an image makes controlling its movements easy.

[FourthWorld]

A page of text can easily be searched to find to position of a specific character.

"Easily"? Have you done it?

It's easy when you're working with textual data. But such pattern recognition in an image is not trivial.

"Commonly" is not "easily". The difference is the availability of packages to handle the task.

Give the wide range of factors that create noise in images and the complexity of accounting for them, I suspect you'll need to find a package for the image recognition part that has features and licensing appropriate for your needs.

Once you find that you may be able to work with it via LC Script, or perhaps via LC Builder, depending on the package's API requirements.

When you find a good package drop the URL here and we can help you find the best way to work with it.

[dunbarx]

Hi.

@ajperks: if I understand what you want, you can loop through all the pixels of a control or image. In this case, a button named "xx":

Code: Select all

on mouseUp
   lock screen
   put the rect of btn "xx" into temp
   add the top of this stack to item 2 of temp
   add the top of this stack to item 4 of temp
   add the left of this stack to item 1 of temp
   add the left of this stack to item 3 of temp
   repeat with y = item 1 of temp to item 3 of temp
      repeat with z = item 2 of temp to item 4 of temp
         set the screenmouseLoc to (y & "," & z)
         put the mouseColor & return after accum
      end repeat
   end repeat
end mouseUp

@ Richard et. al.

I keep forgetting if there is a function, the "pixelColor" which would return the color of any pixel on screen:

Code: Select all

answer pixelColor(x,y

It would sure speed up the process of finding the color a a particular pixel (or all the pixels) in a certain control.. The above requires that the cursor first be placed at the loc of each pixel, so that the mouseColor can be used to find the color at that point.

Craig

[FourthWorld]

@ Richard et. al.

I keep forgetting if there is a function, the "pixelColor" which would return the color of any pixel on screen:

Code: Select all

answer pixelColor(x,y

It would sure speed up the process of finding the color a a particular pixel (or all the pixels) in a certain control.

Controls rendered on screen have much greater predictability than attempting to discern objects in a real-world setting with variances in lighting, shadows, reflections, other objects, and a million other noise factors.

It's not at all impossible. But far from trivial. I would not recommend attempting it from scratch, nor is that needed in a world where people who need this sort of thing rely on packages written by specialists.

[dunbarx]

I may have neglected to explicitly say what to do with this. You asked:

I need to search an arbitrary image, let's say the screen or a JPG for the specific occurrence of a coloured pixel

So in the handler I posted, assuming you knew the RGB value of the pixel of interest, you would:

Code: Select all

repeat with y = item 1 of temp to item 3 of temp
      repeat with z = item 2 of temp to item 4 of temp
         set the screenmouseLoc to (y & "," & z)
         if the mouseColor = yourSpecialColor then
            answer y & "," & z
            exit repeat
      end repeat
   end repeat

instead of just collecting all the colors of each pixel.

But you also asked to be able to search for a specific "mousePointer". What is a "mousePointer"

And did you mean also to be able to determine if a graphic of some sort, of some particular geometry, is present on screen? I can see how to do that as well, built want o know if that is what you really meant.

Craig

[dunbarx]

Richard.

All that. But am I wrong in thinking i remember that there is a direct way to determine the color of a single pixel, without having to jump through hoops to set up the "mouseColor"?

Craig

[FourthWorld]

If the goal involves precessing the entire image, it's often cleaner and faster to walk through every four bytes of the imageData.

Several years ago one of the members here made an an image engagement framework that included some basic filters. It's been so long I no longer recall his name or the thread name, but it used a loop through the imageData. It was as slow as one would expect slogging through so much data in a scripting language (it would be cool to see an image filter external), but it was fun to see the ambition played out well.

[FourthWorld]

FWIW I found the old thread about the image processing toolkit:
https://forums.livecode.com/viewtopic.p ... 490#p45383

[Newbie4]

@dunbarx
If I remember correctly, each pixel has 4 bytes: Alpha, R, G and B. So you can get the color of any pixel with the code

Code: Select all

put  charToNum (char (pixelStart + 1) of image1) into tAlpha                     // byte #1 is the Alpha
put  charToNum (char (pixelStart + 2) of image1) into tRed                       // byte #2 is the Red
put charToNum (char (pixelStart + 3) of image1) into tGreen                     // byte #3 is the Green
put charToNum (char (pixelStart + 4) of image1) into tBlue                       // byte #4 is the Blue 

Is that what you were asking?

[dunbarx]

Newbie4.

Sort of.

I can set the loc of the hot spot of the cursor to a certain pixel, and then use the "mouseColor" to read its value. That is what I posted early on in this thread, But I was wondering if there is a more direct way, something like:

Code: Select all

get pixelColor(x,y) 

In other words, is there anything like that imaginary function? The machine "knows" the color of each pixel. After all, it was the one that drew them all in the first place. It ought to be able to tell me what it already knows.

Craig

[FourthWorld]

@dunbarx
If I remember correctly, each pixel has 4 bytes: Alpha, R, G and B. So you can get the color of any pixel with thecode

Code: Select all

put  charToNum (char (pixelStart + 1) of image1) into tAlpha                     // byte #1 is the Alpha
put  charToNum (char (pixelStart + 2) of image1) into tRed                       // byte #2 is the Red
put charToNum (char (pixelStart + 3) of image1) into tGreen                     // byte #3 is the Green
put charToNum (char (pixelStart + 4) of image1) into tBlue                       // byte #4 is the Blue 

Thank you for posting that example. One updates may be useful:

With the introduction of Unicode support, "char" and "byte" are no longer guaranteed to be synonyms.

You might consider using byteToNum instead of charToNum when working with binary data.

[dunbarx]

Hmmm.

No such animal. eh?

Craig

[FourthWorld]

No such animal. eh?

I'm not aware of one, and checking the See Also list for mouseColor I see nothing like that.

But if it's needed you can write a function to do that in just a few lines using the byte operations from the example above as a starting point.

[dunbarx]

Richard, newbie4.

And how do I get the color of the pixel at "150,450"?

The only way I know is to set the screenMouseLoc and then retrieve the mouseColor. What are you guys doing that is different?

Craig

[FourthWorld]

And how do I get the color of the pixel at "150,450"?

Of the screen, the card, or a particular image?

If you only need one pixel color just use the mouseColor trick.

I'm tempted to write a function for you to illustrate how to use imageData, but it's not as useful if you only need a single pixel value.

What is it you need to do?

And if it isn't related to what the OP created this thread for, can we address your project's need in a thread focused on it?