LiveCode Forums.

Posted: **Fri Feb 19, 2021 7:52 pm**

I want to use the Browser Widget to retrieve data from a collection of similar but different web pages. I want to get the data quickly and reliably. The URLs are contained in the field "URLS". The following code (abbreviated) seems to work.

Code: Select all

on mouseUp pMouseButton
   repeat for each line tUrl in field "URLS"
      set the url of widget "Browser" to tUrl
      wait 60 ticks
      put the htmltext of widget "Browser" into tText
      --parse tText to get the data I want
   end repeat
end mouseUp

The pause between setting the URL for the widget and getting the htmltext is necessary to give the widget time to finish the retrieval from the web page. I know that if I cut the wait to say 10 ticks that the retrieve will not be complete and some if the data I want will not be supplied. I suppose, depending in the web traffic. that my wait of 60 ticks may not be enough on some occasions.

Since I would like to have the retrievals of data be as close as possible (i.e. No wasted ticks) I tried a different approach. I have created a custom variable in the widget called cBusy and made some code modifications to now look like the following.

I added this code to the browser widget:

Code: Select all

on browserNavigateBegin pUrl
   set the cBusy of me to true
end browserNavigateBegin

on browserNavigateComplete pUrl
   set the cBusy of me to false
end browserNavigateComplete

My mouseUp code was changed to this;

Code: Select all

on mouseUp pMouseButton
   repeat for each line tUrl in field "URLS"
      if tUrl="ALL" then next repeat
      set the cBusy of widget "Browser" to false
      set the url of widget "Browser" to tUrl
      repeat
         wait 5 ticks
         put the cBusy of widget "Browser" into tFlag
         if tFlag is false then exit repeat
      end repeat
      put the htmltext of widget "Browser" into tText
      --parse the text to get the data I want
   end repeat
end mouseUp

This executes but only gives me the data for the last website. If I change the wait from 5 ticks to 60 ticks it does work. I put break points in the widgets code at the two places where I have "set the cBusy of me to" and, interestingly enough, in the 60 tick version I has all my answers before any stop. When It finally did break it was at "browserNavigateBegin".

Any ideas?

Thanks,
Larry

Posted: **Fri Feb 19, 2021 8:03 pm**

Do you want to render the page for display to the user, or do download the page data?

The browser widget is great for rendering, but complex overkill for a simple download.

For the latter, see the GET url command.

Posted: **Sat Feb 20, 2021 7:19 pm**

Thanks Richard,

I had no idea there was a get url command. I thought I had to render it first and then get the hymltext for the information I wanted. Will I get the htmltext basically instantly or will I have to do some waiting? There are times when I want to see the page but not for this case.

Larry

Posted: **Sat Feb 20, 2021 7:47 pm**

The repeat loop is blocking which is why you don't see browserNavigateComplete until the loop is finished, and at that point only the last url is being processed. Ideally you'd use "send" instead of the repeat, but you can fix the problem by adding "with messages" to the wait command inside the loop.

But using "get url" will be faster and preferable since no rendering is needed, and also has the advantage that the next line of the repeat loop won't run until the URL has been fetched or the timeout is expired. You don't need a wait, though it would be good to check if a timeout occurred so you can try again.

Posted: **Sat Feb 20, 2021 10:08 pm**

I would also suggest that if you have planned time to wait, then add "with messages" to the end of it, so house keeping can be done while your already waiting. It would look like -

Code: Select all

repeat
         wait 5 ticks with messages 
         -- continue on...

Posted: **Sat Feb 20, 2021 11:35 pm**

lohill wrote: ↑
Sat Feb 20, 2021 7:19 pm
I had no idea there was a get url command.

There's also a POST command for sending data to a server.

Will I get the htmltext basically instantly or will I have to do some waiting?

That's the question every Internet user has.

Round trip time from your sending the request until you receive the reply will vary, depending on your connection speed, your server's connection speed, server processing time to satisfy the request, server load, how many hops each route takes, and more.

Small static pages often take less than a second. IMNSHO if it takes longer than six seconds abandon that server and use anything else.

LiveCode Forums.

Browser Widget Timing

Browser Widget Timing

Re: Browser Widget Timing

Re: Browser Widget Timing

Re: Browser Widget Timing

Re: Browser Widget Timing

Re: Browser Widget Timing