Web Page data - Still trying

lohill · Post by **lohill** » Sun Jun 13, 2010 1:02 am

Bernd,

If I use your script I dont find what you are looking for in the source code of the page. Maybe you access the page with more options? Possibly your Rev script accesses a page with less options.

I think you have hit upon the heart of the matter. You won't even find 'Group Relative Strength'. Why doesn't REV see it when I can see it in my browser and in the view of the source for my browser. Are you or any of the other REV experts aware of anything I can do in REV to get around this obstacle?

I would really be thankful.

Regards,
Larry

P.S. By the way if you have read many of my other posts, you will note that I have been able to see this data while using revBrowser commands. It works fine on my Mac but not at all on my PC.

bn · Post by bn » Sun Jun 13, 2010 1:14 am

Larry,

I think you have hit upon the heart of the matter

It might be that depending on your cookies you have access to a paid content of the site. I dont think that put URL tURL gives access to the cookies/log in.
Unfortunately I dont know if and how one could do that from the put URL command.
If you use rev browser it uses the browser engine of the operating system and behaves like a browser. Maybe that is why you can access the 'other' content.
You could still use the rev browser and get to the html source of the "full" page and do as Andrew proposed and you tried to extent.
I am afraid that I am not a big help with rev browser since I hardly used it up to now.

regards
Bernd

lohill · Post by **lohill** » Mon Jun 14, 2010 8:33 pm

Bernd and Andrew,

I decided to go back to revBrowser to try to solve my problem and I have found what I believe is a bug in revBrowser. I think this needs to be called to the attention of the REV programmers. What is the proper way for doing this?

It is very simple to demonstrate and has to do with how revBrowser works on a PC as opposed to how it works on a Mac. The simple demonstration has you create a stack with a single button whose code is shown below. It is best to place the button near the top right side of the window because of where the rect paints the results of the browse. The URL that is used is one that anyone should be able to access without having to worry about permission.

Code: Select all

on mouseUp
   local tBrowserId
   put "http://finance.yahoo.com/q?s=AAPL"  into tURL
   put revBrowserOpen(the windowId of this stack, tURL) into tBrowserId
   revBrowserSet tBrowserId, "rect", "10,50,500,500"
   if tBrowserId is not an integer then
      answer "Failed to open browser"
      exit mouseup
   end if
   wait 10 seconds
   put revBrowserGet(tBrowserId, "htmltext") into tString
   answer len(tString)
   put tString
   revBrowserClose tBrowserID
end mouseUp

On a Mac the rect is filled with a visible version of the browse, the length of the string tString is answered ( varies but over 60,000 chars) and the string itself is placed in the message box for inspection. Works fine on a Mac and I'm positive you could modify the URL with other stock symbols and it would still work.

On a PC, however, the length of the string is 41 and the message box is filled with:
<HTML><HEAD></HEAD>
<BODY></BODY></HTML>

Visually, when you run it on a PC, you will briefly notice the rect filled with the results of the browse but then a line that says 'Loading...' is inserted directly above where it says 'Apple Inc.- The browse never finishes apparently because the message remains there until everything gets reset. As fair as I know that 'Loading..' message never appears on the Mac or at least doesn't appear long enough to notice. Also 10 seconds should be more than enough time for things to complete.

Please give this a try to make sure I'm not seeing things and then advise me about officially reporting it.

Thanks you and regards,
Larry

mwieder · Post by **mwieder** » Wed Jun 16, 2010 12:39 am

works here on xp.

lohill · Post by **lohill** » Wed Jun 16, 2010 2:47 am

mweider,

Thanks for taking a look? Do you see 'Loading...' on the screen with the PC?
I have used REV 4.0.0 and 4.0.5 and it has worked fine on my Mac. The first PC place I tried was on Windows XP under VM Fusion and it did not work. The second place was Windows 7 on my laptop with REV 4.0.0. It failed too. I don't have any earlier versions of REV to test.

I also ran the Browser Sampler that REV supplies today both on my Mac and PC under VM Fusion. Using the same URL, I never saw 'Loading...' on the Mac but it showed and persisted on the PC.

Regards,
Larry

mwieder · Post by **mwieder** » Wed Jun 16, 2010 5:22 pm

No "loading..." message here. I *did* have problems with cookies - your ten second timeout wasn't enough to handle all the prompts that came up for me to dismiss all the advertising on that site, and so I got only partial html information. But if I increased the timeout to the point where I could actually deal with the dozen or so cookie prompts then I ended up with 5407 chars of htmltext consistently.

lohill · Post by **lohill** » Wed Jun 16, 2010 10:53 pm

mweider,

But if I increased the timeout to the point where I could actually deal with the dozen or so cookie prompts then I ended up with 5407 chars of htmltext consistently.

That is interesting. On my Mac, I can cut that wait down to a second and never seem to miss getting all the htmltext. By the way, I think you should be getting at least ten times as much htmltext as you are getting. As I said earlier, I get well over 60,000. I'll try some longer waits on my PC but even if it takes 12 seconds that is way too long to wait.

I don't know if the revBrowserGet "busy' commnand works but the following code on my Mac gets the correct result with i = 1. On the PC, i gets to 1000 and still no result.

Code: Select all

on mouseUp
   local tBrowserId
   put "http://finance.yahoo.com/q?s=AAPL"  into tURL
   put revBrowserOpen(the windowId of this stack, tURL) into tBrowserId
   revBrowserSet tBrowserId, "rect", "10,50,500,500"
   if tBrowserId is not an integer then
      answer "Failed to open browser"
      exit mouseup
   end if
   --wait 10 seconds
   repeat with i = 1 to 1000
      if revBrowserGet(tBrowserId, "busy") = false then exit repeat
   end repeat
   answer i
   put revBrowserGet(tBrowserId, "htmltext") into tString
   answer len(tString)
   put tString
   revBrowserClose tBrowserID
end mouseUp

Regards,
Larry

mwieder · Post by **mwieder** » Wed Jun 16, 2010 11:20 pm

We've been through this before. You have different cookies set for your access.

If I were doing the data mining I wouldn't use revBrowser, I'd just collect the htmltext directly. You don't have any control over http timing using the browser.

lohill · Post by **lohill** » Thu Jun 17, 2010 1:06 am

mweider,

We've been through this before.

You can say that again -and again -and again!

I'd just collect the htmltext directly.

Do you mean via 'put URL tUrl into tString'? I have been there multiple times also and no one is willing to help me with the cookies. The data I have gotten with this approach does not include the specific piece of data I need - namely Group Relative Strength. RevBrowser has shown me the most promise, at least for the Mac. Unfortunately, there are some PC users I want to be able to use this.

I am still convinced that there is a bug in revBrowser because I can visually see the data on the PC bur it does not show up in the htmltext.

Thanks for your efforts in trying to help me.

Regards,
Larry

mwieder · Post by **mwieder** » Thu Jun 17, 2010 1:22 am

You can say that again -and again -and again!

rotfl

Do you mean via 'put URL tUrl into tString'?

Well, yes, that's what I did have in mind. I have by now <sigh> forgotten what your original intent was for this system. I didn't realize this was for other users.

But aren't you going to have a problem with other users using this system? Everyone won't have the same cookies you do. I assume you have to log into the site. Can you put your user authentication into the url?

Code: Select all

put url "http://user:password@finance.yahoo.com/q?s=AAPL"  into tURL

LiveCode Forums.

Web Page data - Still trying

Re: Web Page data - Still trying

Re: Web Page data - Still trying

Re: Web Page data - Still trying

Re: Web Page data - Still trying

Re: Web Page data - Still trying

Re: Web Page data - Still trying

Re: Web Page data - Still trying

Re: Web Page data - Still trying

Re: Web Page data - Still trying

Re: Web Page data - Still trying