How can an image be searched?

[dunbarx]

Richard.

Thank you, but I do not need it at all. I used the mouseColor "trick" to help the OP. I was just wondering if there was such a thing. It seems like a natural.

Craig

[jiml]

If the requirement is to get the colors of many pixels on the screen it is rather slow to keep setting the screenMouseLoc and grabbing the mouseColor.
It would be faster to just import screenshot and explore the pixel colors via imageData.

Jim Lambert

[dunbarx]

OK.

Is it the imageData that contains the RGB/Channel data in an array or list of some kind? Is that the source Newbie was talking about?

Because the way the mouseColor "trick" has to work, if you examine my early post, is to set up an x/y scan, pixel by pixel, within the rect of the control. And that is slow.

Craig

[FourthWorld]

Is it the imageData that contains the RGB/Channel data in an array or list of some kind?

The imageData contains the binary bytestream of the rendered image, where every four bytes describe one pixel (aRBG). Being binary is why byteToNum is needed to transform each value into an integer we can then use LC's comparison and arithmetic operators on.

The Dictionary describes this in better detail than I can here.

A fun exercise might be to write a function that return the comma-delimited integers from a given pixel in an image (tip: the bytestream has no concept of x or y, so you'll need to obtain the object's width and account for that when you determine which four bytes to extract and transform).

[dunbarx]

Richard.

Interesting.

This would have no application to, say, a button, which has no such concomitant info.

Anyone ever want to find the color of one or more pixels in general? If I get even one reply, I will make a feature request to the team to have the mysterious "pixelColor(x.y)" function developed. Again, the machine drew them; it ought to be able to describe them.

In other words, it ought not to be required to set the cursor hotSpot on a certain pixel in order to find out the color of that pixel. Why not just ask for the color of that pixel??

Craig

[ajperks]

Thank you all for your suggestions.
The setup I envisage would be an immoveable webcam that covers a set area of a room.
Within the room would be the robot and it would have a point of light beacon on the top, emitting a unique colour.
As you might imagine, as the robot moves, so does the position of the light spot.
I have yet to examine what the imagedata looks like, but if the image consists of rows and columns, like the pixels in a displayed image (magnified of course) then the count along the row, and count down the column, would be equivalent to the x-y axis of a map.
For convenience, the image would be on screen, so is there any code that can scan row and column looking for the unique colour. Note, the colour may be spread over several pixels but it only needs to detect the first occurrence.
Am I wishing for the impossible?
If I was clever, I would use the OpenCV or Tensorflow libraries, but I am not. Apart from the fact that there is always some bit if the system that will not download or install like it should. Better to use an axe to cut down the tree if the chainsaw won't work, I say. That was supposed to be amusing...

[FourthWorld]

This would have no application to, say, a button, which has no such concomitant info.

Everything on screen is ultimately a bitmap. You can obtain the bitmap of a control with the "export screenshot" command.

Anyone ever want to find the color of one or more pixels in general? If I get even one reply, I will make a feature request to the team to have the mysterious "pixelColor(x.y)" function developed.

Wanting <> needing. I want everything; I don't expect the team to build everything, esp. things I can write in script with modest effort using the features they've already provided.

If you need this you can write this function in just a few minutes.

If you don't need it why require the engine team take the time to consider it?

A good rule of thumb for submitting feature requests for any software is:

What development effort already in queue would you like to see them set aside in order to accommodate the new request?

[FourthWorld]

The setup I envisage would be an immoveable webcam that covers a set area of a room.
Within the room would be the robot and it would have a point of light beacon on the top, emitting a unique colour.
As you might imagine, as the robot moves, so does the position of the light spot.
I have yet to examine what the imagedata looks like, but if the image consists of rows and columns, like the pixels in a displayed image (magnified of course) then the count along the row, and count down the column, would be equivalent to the x-y axis of a map.
For convenience, the image would be on screen, so is there any code that can scan row and column looking for the unique colour. Note, the colour may be spread over several pixels but it only needs to detect the first occurrence.
Am I wishing for the impossible?
If I was clever, I would use the OpenCV or Tensorflow libraries, but I am not.

It requires less cleverness to use the work of specialists than to replicate it from scratch.

There are at least two initial challenges:

1. A single camera can tell you the lateral position of an object, but determining the other axis -- how far away the object is from the camera -- is tricky. One-eyed humans have difficulty estimating distance; machines more so.

2. Cognitively, a given light source can be described as a specific color. But the physics of light and vision reveal a murkier reality, where what's emitted can be perceived with at least slight variation depending on noise factors (clarity, influence from ambient light, etc.).

You might begin this exercise with an experiment:

Take a photo of your room with a distinctive light emitter in it, bring it into your computer, zoom in very close to examine the pixels.

That will show you the difference between how we conceptualize color and how colors are recorded in digital media.

Then turn on any other light in the room, take a second picture, and examine those pixels.

That will illustrate the effects of noise.

This is not a trivial problem. If it were me I'd rely on available work from specialists who've already devoted years to the problem.

[dunbarx]

Great stuff, and Richard points out lots of real world pitfalls.

But if the idea is to find the first of certain pixels with a certain color, is there any other way than using the "mouseColor trick"? Because I thought I understood that the imageData does not map to pixel x/y values in any way. If this is in fact true, then the trick should work fine, though would hardly be speedy.

Another issue is this. What color are we looking for? It is not likely to be any sort of fixed RGB value, but rather one within a range of those values. This is not really a problem, since one only has to search "sloppily" (pseudo):

Code: Select all

 put the rect of yourScreenImage into temp
repeat with y = item 1 of temp to item 3 of temp
      repeat with z = item 2 of temp to item 4 of temp
         set the screenmouseLoc to (y & "," & z)
         get the mouseColor 
         if abs(item 1 of it - item 1 of yourDesiredColor) < 10 and abs(item 2 of it - item 2 of yourDesiredColor) < 10 and\
           abs(item 3 of it - item 3 of yourDesiredColor) < 10 then
           answer  y && z
           exit to top
         end if
      end repeat
   end repeat

Craig

[dunbarx]

Rereading some of the replies here, I get the feeling that there is in fact a way to map a line in the imageData into an x/y value within an image. So that once a line in the imageData is picked out as an acceptable match to the desired color , it can locate that pixel in the actual image.

Is this so? And if so, the speed issue goes away.

Craig

[SparkOut]

I don't know what the desired test for the robot's position is, but it strikes me that if the check is for "when the robot reaches the desired location then..." it would be easy to determine the binary data "row" and "position within that row" (x and y) for that location in the static image file. You can then restrict the check for pixel colour within the data file, rather than compare all pixels.

Of course, that's not a help if the idea is determine where the robot is within the image at all times. I'd definitely agree with Richard on this.

[dunbarx]

Sparkout.

it would be easy to determine the binary data "row" and "position within that row" (x and y)

So i just missed all that. The binary data does indeed map the pixels in an x/y manner.

Craig

[ajperks]

It was mentioned that judging the distance the robot was from the camera would be hard to calculate. It isn't an issue, because the camera would be looking down, not along the robot path. The room extremities would become the edge of the map.
Just imagine a camera fixed to the ceiling and capturing the whole of the floor.
The idea is to see the beacon light point in this area.
Ignore scale and ambient lighting for now. Perhaps there is no room lighting, for the purpose of this application.

I haven't actually seen confirmation that the x,y value can be determined, at all, by the software. That is crucial to the problem. So, whatever spin, possible side issues, practicality, alternative software systems you might come up with, I am only interested in how the light point can be detected and related to some x,y position by livecode.
There has been mention of mouse position in answers. The mouse has no relevance to the problem because the mouse will never be used or connected.
Can livecode determine the x,y position of the light point in the camera image? (which might be the screen or a JPG, whatever gives the answer.)

[dunbarx]

ajperks

The "trick" I used does not actually require a mouse. The property "screenMouseLoc" sets the pointer. This pointer exists regardless of whether an actual mouse exists. Once the pointer is set, the pixel below its hot spot can be read.

it appears to me that the robot's light is very different in both color and brightness from any other feature in the room. Because of that, it ought to be simple and reliable to find it.

Craig

[dunbarx]

It further occurs to me that, if indeed the robot's "headlight" is radically different than any other feature in the room, it is possible to use my original handler, but only test pixels that are, say, 10 apart. That will give a 100-fold increase in speed.

The penalty for that speed is accuracy. If you really need single pixel resolution, then never mind. But ten pixels may translate to a very small physical distance in the room itself.

So what sort of accuracy do you require? You can always check every fifth, third, whatever pixel, and the speed increase will still be marked.

Craig