Web Page data - Still trying

[lohill]

I'm still trying to get this web page to deliver all its data. By using the Live Headers addon for FireFox, I have discovered the critical cookie that should make me an authorized user of the data on this page:http://www.investors.com/StockResearch/ ... ymbol=AAPL
It is called .ASPXAUTH and I have used libUrlSetCustomHttpHeaders to splice it into a client request to get the data. The server is not happy with the request as you can see from the log excerpt below. Can any one suggest changes that would get me closer to my solution?

Thanks,
Larry

socket selected: 63.71.211.170:80|6927
GET /StockResearch/Quote.aspx?symbol=AAPL HTTP/1.1
Host: http://www.investors.com
User-Agent: Revolution (MacOS)

HTTP/1.1 200 OK
Cache-Control: public,no-cache,no-store,max-age=0,must-revalidate,proxy-revalidate
Date: Mon, 26 Apr 2010 22:37:20 GMT
Content-Length: 157037
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/6.0
SID: 19
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
CommunityServer: 4.0.30417.1769
Set-Cookie: CSUserCookie=2101; domain=.investors.com; path=/
Set-Cookie: CommunityServer-UserCookie2101=lv=Fri, 01 Jan 1999 00:00:00 GMT&mra=Mon, 26 Apr 2010 15:37:20 GMT; expires=Tue, 26-Apr-2011 22:37:20 GMT; path=/
Set-Cookie: CommunityServer-LastVisitUpdated-2101=; path=/
Set-Cookie: CommunityServer-UserCookie2101=lv=Fri, 01 Jan 1999 00:00:00 GMT&mra=Mon, 26 Apr 2010 15:37:20 GMT; expires=Tue, 26-Apr-2011 22:37:20 GMT; path=/
Set-Cookie: ASP.NET_SessionId=2p0sueypmuzv0d45dbgig245; path=/; HttpOnly
Set-Cookie: IBDCampaigns=freetrial=freetrial|1|4/26/2010 3:37:20 PM|A; domain=.investors.com; expires=Tue, 26-Apr-2011 22:37:20 GMT; path=/
Pragma: no-cache
Expires: Mon, 26 Apr 2010 22:37:20 GMT

socket selected: 63.71.211.170:80|6927
GET /StockResearch/Quote.aspx?symbol=AAPL HTTP/1.1
Cache-Control: public,no-cache,no-store,max-age=0,must-revalidate,proxy-revalidate
Date: Mon, 26 Apr 2010 22:37:20 GMT
Content-Length: 157037
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/6.0
SID: 19
X-Powered-By: ASP.NET
X-AspNet-Version: 2.0.50727
CommunityServer: 4.0.30417.1769
Cookie: .ASPXAUTH=6D904726BC8A781EB46F79BF1450A25CAEB87D79D348DBEE84CF5077FE1AD79F27EF536587D4FA4E14AB817CFF5C071F5BB6D3F002C302DED1A308A7C6AEE2F6E6D7CAF92DDC2A1B4AEEF5644E5FC730114A4B8D211C2B0C51A1EDF1DE17368DDE70D8A2EAC1638CEE8FDF87F7BF04D27B410941D35D5898319762359191BBBBCDB5B5A2450231AFD8C7E671FE6F59CCAF22D52450065E61CBCF2D9DC0A495EC226E8BFBE467DF919C8DF7F6FD649BC3FA7791EC768533BA0A8A089489E402CA43C771F8B16D50F27CFB1F9846E4498FF55F3758F25BA6B7977068121EC4B00AA164A77D4F4723E0891CF4B14EC6140B9EBBC977A8CABA393621EBA5DC0FDAA34D576DAFAB45072848409C5F9B0AC060E75A47E458F07528BB725DA5613E0A03D7103565A9F159EFEC9EA54DABEDCC08F428B96C2722185F38C1A1DA236DDB1F; CSUserCookie=2101; CommunityServer-UserCookie2101=lv=Fri, 01 Jan 1999 00:00:00 GMT&mra=Mon, 26 Apr 2010 15:37:20 GMT; CommunityServer-LastVisitUpdated-2101=; CommunityServer-UserCookie2101=lv=Fri, 01 Jan 1999 00:00:00 GMT&mra=Mon, 26 Apr 2010 15:37:20 GMT; ASP.NET_SessionId=2p0sueypmuzv0d45dbgig245; IBDCampaigns=freetrial=freetrial|1|4/26/2010 3:37:20 PM|A; Pragma: no-cache
Expires: Mon, 26 Apr 2010 22:37:20 GMT

HTTP/1.1 400 Bad Request
Content-Type: text/html
Date: Mon, 26 Apr 2010 22:37:21 GMT
Connection: close
Content-Length: 20

[dickey]

Hello Larry,

Just to clarify, are you attempting to extract the values from the area of the web page as detailed in the image below?

If so, that is relatively easy by walking through the page source and moving between tags to extract the relevant values.

In this case if you visit the link and right mouse click, and view source, the relevant data is placed between tags:

<!-- begin subscriber view --> and <!-- end subscriber view -->, header details and;
<!-- begin visitor view --> and <!-- end visitor view -->, stock quote details;

You can then walk through the source in a few steps narrowing focus as you extract the relevant data

Code: Select all

local tBrowserId 
put revBrowserOpen(the windowId of this stack, "http://www.investors.com/StockResearch/Quote.aspx?symbol=AAPL") into tBrowserId
revBrowserSet tBrowserId, "rect", "0,0,800,800"
if tBrowserId is not an integer then answer "Failed to open browser"
wait 15 seconds
   
-- Larry sometimes you might need to script data entry to populate fields on a form
--get revBrowserExecuteScript(tBrowserId, "document.form1.cbPO.checked = true;")
--get revBrowserExecuteScript(tBrowserId, "document.form1.drpState.selectedIndex = 1;")
--get revBrowserExecuteScript(tBrowserId, "document.form1.txtSuburb.value = 'Wollongong';")
--get revBrowserExecuteScript(tBrowserId, "document.form1.txtPostcode.value = '2500';")
-- and Larry sometimes you might need to script submitting a form 
--get revBrowserExecuteScript(tBrowserId, "document.getElementById('btnSearch').click();")

-- get html text of result page
wait 5 seconds
put revBrowserGet(tBrowserId, "htmltext") into fld "fldURLString"
put the text of field "fldURLString" into t_string


put offset("<!-- begin visitor view -->",t_string) into t_startMarker1
put offset("<!-- end visitor view -->",t_string) into t_endMarker1
put char t_startMarker1 to t_endMarker1 + 25 of t_string into fld "fldResultString"

Larry, just keep drilling down by tag until you get the values you require. In the end (probably take about 30-60 mins) first time around to script, but you should end up with a method to post any stock code and extract the quote data as required.

I hope this helps.

Kind regards,

Andrew

[lohill]

Andrew,

Thank you very much! I had given up trying but you have put me onto another tack that looks very promising. My problem was a result of trying to get data from a web page that required a logon. Even though my browser was set to automatically recognize me as a valid user, the things I was trying with REV wanted more authorization. I had noticed however that the demo of revBrowser seemed to let me get to where I wanted to go just fine but I didn't know how to scrape the information off the page.

Your reply showed my how and if I make the 'rect' for the browser real small, the user will never know what I am doing.

Thanks again.
Regards,
Larry

[dickey]

That's great Larry, I am glad you noticed my reply (I thought it was going to pass down the list and disappear),

If you need any help with the script, just let me know the exact items you wish to extract. I don't mind spending 30 mins getting a working example completed for you to follow next time around.

You can, if you watch a number of stocks, script the entry of stock codes etc, before you extract the quote. The sample code in my first email has an example of that technique.

That way you could start with a list of say 10 stocks - obtain a quote for each - extract the values for each stock, then push the results to a database or datagrid etc.

Good luck with the project.

- Andrew

[lohill]

Andrew,

This is what I have ended up with for my project. It seems to work but only time will tell. It is a function that I use to get a particular piece of data that is not always present in eIBD but can be found in StockCheckup. I give it a ticker and it comes back with a number. Here is what it looks like:

Code: Select all

function GrsFromCheckup tTicker
   local tBrowserId
   put "http://www.investors.com/StockResearch/StockCheckup.aspx?symbol=" & tTicker into tURL
   put revBrowserOpen(the windowId of this stack, tURL) into tBrowserId
   revBrowserSet tBrowserId, "rect", "0,40,10,50"
   if tBrowserId is not an integer then
      answer "Failed to open browser"
      return "Error"
   end if
   Repeat with i = 1 to 10
      wait 1 second
      put revBrowserGet(tBrowserId, "htmltext") into tString
      put offset(">Industry Group Rank",tString) into tStart
      if tStart > 0 then exit repeat
   end repeat
   If tStart > 0 then
      put char tStart to tStart + 250 of tString into tResult
      put offset("</td>",tResult) into tStart
      put char tStart+6 to -1 of tResult into tResult
      put offset("</td>",tResult) into tStart
      put char 1 to tStart-1 of tResult into tResult
      put last word of tResult into tResult
      return tResult
   else
      answer warning "Error connecting to StockCheckup."
      return "Error"
   end if
end GrsFromCheckup

There are lots of parts about it that I don't understand - like the significance of 'rect' and what it is. It is visible but not really an object as nearly as I can tell. I make it really small so the user can't see it. I'm also not sure how long the 'wait' needs to be so I have put it in a finite loop. As you also can see, I don't know much about web page structure but I can parse any chunk of text.

Looking at the other sample code you sent, I can see I need to spend some time studying revBrowser. Are the Dictionary and the User Guide for REV the best places to start? Are you saying that with a command like revBrowserExecuteScript that I can pretty much enter data into web pages (for example login) and navigate around and get whatever I need all within the REV environment.

My current project is for a group of volunteers that gather data from eIBD each week and prepare a list which is distributed to the rest of the group. We have been doing it with Excel and macros. So right away that lets out all the Mac people. Now there will be a REV application that does the job for both types of machines and it will be a lot easier for the novice to run.

If revBrowser is as powerful as I think you are saying, I have another job that I volunteer for that involves getting information on dogs in our local animal shelter (from the web site) and entering it into the PetFinder web site. I do that currently with Excel but it takes some effort that might be able to be simplified from within REV. Any way it will be fun trying.

Thanks again for your help. I hope you'll be listening for my pleas when I get stuck.

Larry
P.S. Any suggestions on the above code will be appreciated.

[lohill]

Andrew,

On further review... That function I devised above seems to work fine on a Mac but in Windows (actually XP under VM Fusion) it seems to get around the loop 10 times never having found the data. I even changed the wait to 10 seconds. Any clue?

Regards,
Larry

[dickey]

Hello Larry,

I will now attempt to answer some of your questions:

There are lots of parts about it that I don't understand - like the significance of 'rect' and what it is.

I believe you are referring to the code from my example

Code: Select all

revBrowserSet tBrowserId, "rect", "0,0,800,800"

The prototype syntax for revBrowserSet is revBrowserSet instanceId, propertyName, propertyValue. In the example we are setting the value of the 'rect' property which the dictionary refers to as:
"rect" : The rect of the browser object. The default is 0,0,0,0 so it is necessary to set the rect of a newly created browser before it will appear to be visible. So basically Larry, if you want to see the browser window, you need to set values for the left, top, right, bottom position of the browser window.

...next question

I'm also not sure how long the 'wait' needs to be so I have put it in a finite loop.

I agree this is a little confusing. In Rev there are two sets of browser commands - the older XBrowser_ syntax (which seemed to have greater functionality than the newer), and the newer revBrowser syntax. The use of wait in my example was upon reflection inappropriate. I should have trapped on browserDocumentComplete pInstanceId, pUrl to verify that the target web page had finished loading.

...next question

Are you saying that with a command like revBrowserExecuteScript that I can pretty much enter data into web pages (for example login) and navigate around and get whatever I need all within the REV environment.

Yes and no. Using revBrowserExecuteScript you can automate data entry and submit web forms, and by writing a custom page reader extract any values you wish to extract from the result page, in fact any page, with several exceptions. RevBrowser is not a native Rev object and therefore (to the best of my knowledge and testing) cannot receive focus, and therefore you cannot use direct to screen techniques like type and click. This makes it unsuitable for use with websites authored in Flash or Silverlight. From what you have described and the example sites you have provided you shall be fine. I am currently writing a program in Rev that will allow you to highlight the objects you wish to extract from a web page, have the data entry / extract saved to a schedule and written to a database automatically regardless of the code of the page. I will keep you posted on that as I go.

...next question

I have another job that I volunteer for that involves getting information on dogs in our local animal shelter (from the web site) and entering it into the PetFinder web site

So basically, you gain a list in excel format and manually key that into a web based form on PetFinder. No problem. You can have Rev open a .csv file, then open a web browser and key the entries and submit them automatically for you. I recently used this method to process 10's of thousands of records to a new accounting system. No typing errors (that is if the source records are ok). To see a very basic example of this, please see below:

I created a fake password page @ http://www.buriedtreasure.com.au/fakesecure.php , the source of that example page is as follows:

Code: Select all

<?php
  $script = $_SERVER['script'];
  if ($_POST['user'] and $_POST['pwd'])
  {
     echo 'your user name is '.$_POST['user'].'</br>';
     echo 'your password is '.$_POST['pwd'];
  }
?>
<html>
  <body>
  <form name="form1" action="<?PHP echo $script; ?>" method="post">
    <input id="user" type="text" name="user">
    <input id="pwd" type="password" name="pwd">
    <input id="submit" type="submit" value="submit">
  </form>
  </body>
</html>

Basically a demo authetication form posting back to itself and outputting the posted variables to screen as proof.

Now if you open Rev and create a button with the following on mouseUp you can test for yourself.

Code: Select all

on mouseUp
local tBrowserId
  
   put revBrowserOpen(the windowId of this stack, "http://buriedtreasure.com.au/fakesecure.php") into tBrowserId
   revBrowserSet tBrowserId, "rect", "0,0,800,800"
   if tBrowserId is not an integer then answer "Failed to open browser"
   wait 10 seconds
  
   -- populate fields
   get revBrowserExecuteScript(tBrowserId, "document.form1.user.value = 'demoUser';")
   get revBrowserExecuteScript(tBrowserId, "document.form1.pwd.value = 'demoPassword';")
   get revBrowserExecuteScript(tBrowserId, "document.getElementById('submit').click();")
end mouseUp

Larry, just a username and password form but it demos the point.

I hope that helps a bit more, I am just about to have a look at your code now, and will post again later.

Kind regards, Andrew

[lohill]

Thanks Andrew,

That example is really slick and it demonstrates the potential. To let you know how naive I am, I have to ask what language are you using in the first set of code? If the form that I want to enter data on is already on the web I would assume I do not need to know that language before being able to script REV to fill it out. At most I would need to be able to identify the element IDs.

Best Regards,
Larry

[dickey]

Hello Larry,

That example is really slick and it demonstrates the potential. To let you know how naive I am, I have to ask what language are you using in the first set of code? If the form that I want to enter data on is already on the web I would assume I do not need to know that language before being able to script REV to fill it out. At most I would need to be able to identify the element IDs.

Larry, no question is a dumb question. In answer to your first question: the example web page was written in PHP. In answer to your second question: (Yes) For html elements you only need to identify elements by their ids (given they have one). For Flash, Silverlight etc it becomes a little more complex.

Kind regards,

Andrew

[dickey]

Hello Larry,

I have written a sample app (stack) to demonstrate extracting data by html tag from a web site. It is a .rev file (not a standalone), therefore you can review the code if you have Rev Studio or Enterprise.

It is available for download at http://www.buriedtreasure.com.au/larry_extract.rev . The stack has been tested for Mac and Windows.

The example stack uses the web site you listed as the target for data extraction. The program extracts a range of values in an automated fashion.

For demonstration purposes data is output as comma delimited text to a scrolling text field (bottom section of screen) and pushed to a data grid (top section of screen).

You can clear the data displayed in the data grid or text field at any time by using the buttons provided.

1. key in the stock codes you wish to extract one stock code per line. I have provided ten example codes - the top ten US stocks by market cap).
2. the scheduler is launched when you run the 'Market Update' script and the value of the schedule is set to a number. You can change or disable the schedule at any time by selecting 'disable'. Please note the schedule is set to disable by default.
3. press the 'Market Update' button to start the show. The button 'Market Update Backup' is there for you to experiment with and is a copy of the button 'Market Update'.

I have demonstrated several different techniques for extracting by html tag in the source of the 'Market Update' button, therefore you should be well equipped to start over with other examples.

The example is comprehensive and the scripts are commented.

Given you are not entering data (your query is a part of the URL string), I did not need to utilise revBrowser (as previously discussed). There are many other times where it is appropriate.

Please advise when you have a copy of the stack as I will then disable the temporary link.

Kind regards, Andrew

[lohill]

Thanks Andrew,

I think I have it but haven't had time to study it yet. A regular click on the link took me to the browser and mostly what I saw was gibberish with some recognizable code interspersed. A right click, though, gave me a chance to download from the link and now I have something that looks like rev source.

Thank you very much for your efforts. I am excited to take a look at it and see where I have been going wrong. I'll probably take a couple of days to respond.

Best regards,
Larry

[lohill]

Andrew,

Wow! This represents a lot of work. I hope it was something you already had and that you didn't have to start from scratch. If you did start from scratch, I'm really impressed. It is going to take me more time than I thought to make sense of this. I see a number of things right off the bat that indicate the way you work that differ from the way I do things.

First of all your commenting is great and something I tend to slight. Thank you.

Secondly your use if the 'backup' button is not something I have used. I have been tending to put most of my major code in either the stack or card script and then having not much more that a call to that script in the button. When I have needed to make a 'backup' script while I am working on something I generally have been copying the original routine or function (on Whatever) and just changing the name to WhateverOLD. Of course that way I have to either comment the whole thing out or make minor changes in the exits or returns. If my new version is successful then I go back and delete everything with OLD in the title.

I like your use of fldStatus. Do you usually make accommodations for resizing the window and the visibility of that field? I have found it very difficult to work with the geometry of resizing windows. I can get pretty close but it always seems to surprise me with changes in appearance that are not expected and that mess up the aesthetics of the window.

The things I am going to particularly have to study are how you choose what to use for tOffsetStart and tOffsetFinish so that I can use offset to help me narrow down on what I want. This is what I believe will take some time and gaining of experience for me.

Also, right at the beginning where you code:

Code: Select all

   put URL tURL into tString
   
   -- in the event this script cannot ***fully*** retrieve the web page
   -- recommendation:
   -- 1. test for the presence of a non dynamic portion of the html <title> tag
   -- 2. test for the presence of a unqiue or significant tag beyond our final extraction point. This latter point is critical if a page is only partially loaded
   put "Stock Quotes, Stock Ratings and Stock News - IBD - Investors.com" into tTitleTag
   put "</html>" into tClosingTag
   if (offset(tTitleTag, tString) = 0 and offset(tClosing, tString) = 0)
   then
      answer error "Error: Unable to retrieve target web page, please try again later..."
      put "Process terminated, URL source failed to fully load." into fld "fldStatus"
      exit mouseUp
   end if

I am going to have to spend some time with your items 1 and 2 for testing. This whole process started for me when I was not able to pull the 'Industry Group Rank' out of the following URL:
http://www.investors.com/StockResearch/ ... ymbol=AAPL

I was consistently able to to find the RS Rating but never the Industry Group Rank. It was like the URL was only giving me part of the information on the page. If you have a quick and dirty solution to that one it would greatly be appreciated.

Thanks again for your help.
Regards,
Larry

[lohill]

Andrew,

I have a series of philosophical question about a couple of lines of your code:

Code: Select all

   -- build the URL from which we shall extract data
   put "http://www.investors.com/StockResearch/Quote.aspx?symbol=" & tStockCode into tURL
   -- place the page source of our URL into a string
   put URL tURL into tString

Who is actually responsible for putting the data into tString?
Ostensibly, it is the REV command URL. But what actually happens?
Is what get put into tString influenced by the user's default browser?
Do the settings in the browser affect what gets placed in tString?
How does URL know whether or not it is even allowed access to the data?
If the default browser of the user has access to the data should that mean that REV should be able to get it too?

Thanks in advance,
Larry

[lohill]

Andrew,
Using a modification of your code, I came up with this simple thing to retrieve the Group Relative Strength from the same page you were working with:

Code: Select all

on mouseUp
   put field "Ticker" into tTicker
   -- build the URL from which we shall extract data
   put "http://www.investors.com/StockResearch/Quote.aspx?symbol=" & tTicker into tURL
   -- place the page source of our URL into a string
   put URL tURL into tString
   
   -- in the event this script cannot ***fully*** retrieve the web page
   -- recommendation:
   -- 1. test for the presence of a non dynamic portion of the html <title> tag
   -- 2. test for the presence of a unqiue or significant tag beyond our final extraction point. This latter point is critical if a page is only partially loaded
   put "Stock Quotes, Stock Ratings and Stock News - IBD - Investors.com" into tTitleTag
   put "</html>" into tClosingTag
   if (offset(tTitleTag, tString) = 0 and offset(tClosing, tString) = 0)
   then
      answer error "Error: Unable to retrieve target web page, please try again later..."
      put "Process terminated, URL source failed to fully load." into fld "fldStatus"
      exit mouseUp
   end if
   
   put "<a class=" & quote & "glossDef" & quote & " href=" & quote & "javascript:void(0);" & quote & " rel=" & quote & \
          "Term.axd?term=Industry Group Relative Strength Letter Rating (Group RS)" & quote & ">" into tSearch
   --answer tString
   put offset(tSearch,tString) into tResult
   
   answer tResult
end mouseUp

Execution of this shows that tResult is 0 indicating the absence of the string. In fact if the line that says 'answer tString' is allowed to run, Group RS is not shown in the visual. Only RS Rating is shown.

However, if I go to the web and view the source code for the same URL I can clearly see the following:

Code: Select all

            <td class="type">
                <a class="glossDef" href="javascript:void(0);" rel="Term.axd?term=Industry Group Relative Strength Letter Rating (Group RS)">
                    Group RS Rating</a>
            </td>
            <td class="rating">
                <span>
                    B-</span>
            </td>

This is why the previous philosophical questions.

Best Regards,
Larry

[bn]

Larry,

make a little stack with one field and one button. The field rather big.
Then put this script into the button:

Code: Select all

  -- build the URL from which we shall extract data
   put "http://www.investors.com/StockResearch/Quote.aspx?symbol=aapl" into tURL
   -- place the page source of our URL into a string
   put URL tURL into field 1

now look if your find the

<a class=" & quote & "glossDef" & quote & " href=" & quote & "javascript:void(0);" & quote & " rel=" & quote & \
"Term.axd?term=Industry Group Relative Strength Letter Rating (Group RS)" & quote & ">" into tSearch

in that field.
If I use your script I dont find what you are looking for in the source code of the page. Maybe you access the page with more options? Possibly your Rev script accesses a page with less options.
The only "glossDef" I find in the source code of that page is:

<a class="glossDef" href="javascript:void(0);" rel="Term.axd?term=EPS Rating">

That is maybe why you dont find it with the script.

regards
Bernd