GetHTTPDataProcedure

Discussion in 'Pascal Logic Code Examples' started by DoctorZoidberg, Aug 18, 2010.

  1. DoctorZoidberg

    DoctorZoidberg

    Joined:
    Aug 8, 2010
    Messages:
    12
    Likes Received:
    0
    Location:
    The Year 3000
    Hi All,

    new to the forum just wondering if anyone has a example of the GetHTTPDataProcedure. Been trying it out but cant seem to get it going....

    I have been playing around with this function, I had a look over the instructions on the help file.
    I can get data from the website but only get the header

    I use as the example
    ReadHTTPData(DataString);

    I get back only header of webpage eg

    <html>
    <head>
    <meta http-equiv="Conten.

    The String variable only holds 50 characters?? How do you increment the readhttpdata?.....

    Thanks

    Zoidy
     
    DoctorZoidberg, Aug 18, 2010
    #1
  2. DoctorZoidberg

    Matty

    Joined:
    Oct 15, 2004
    Messages:
    131
    Likes Received:
    0
    Hi DoctorZoidberg,

    The logic help file has an example as follows for how to use this under the topic 'ReadHTTPData'.

    // get the data
    GetHTTPData('http://MyWeatherForecast.com/index.html');
    // wait for data to come back
    delay(5);
    // read the data into a string
    ReadHTTPData(DataString);
    if DataString <> '' then
    begin
    // Find where the data starts and ends
    DataStart := pos('forecast is', DataString) + 12;
    DataEnd := pos('</body>', DataString);
    // extract the data into the string ForecastString
    Copy(ForecastString, DataString, DataStart, DataEnd - DataStart);
    end;

    There must be a delay between calling GETHTTPData and calling ReadHTTPData.

    By default a string will only be 50 characters long. The return HTTP data will be much longer. In the var section of the logic engine do the following: DataString: array[0..255] of char
    This will give you a string 255 characters long. This can be made much longer if necessary.

    Regards,

    Matty.
     
    Matty, Aug 18, 2010
    #2
  3. DoctorZoidberg

    DoctorZoidberg

    Joined:
    Aug 8, 2010
    Messages:
    12
    Likes Received:
    0
    Location:
    The Year 3000
    Thankyou Matty,

    Setting to array was the problem, other than that and a few tweaks I now have website data being imported correctly.

    Now what to control using it .. :D
     
    DoctorZoidberg, Aug 18, 2010
    #3
  4. DoctorZoidberg

    Garfield

    Joined:
    Dec 3, 2004
    Messages:
    24
    Likes Received:
    0
    Location:
    Adelaide
    My C-Bus enabled rain detecting chicken

    Have you ever been playing around with your robot chicken and not noticed it was raining until your washing on the line was soaked? Unfortunately this happened to me so I decided to have a shot at using the GetHTTPData logic function as well.

    The first step was to C-Bus enable my robot chicken.
    I located the wires that start and stop the dance routine.

    1.jpg

    2.jpg

    3.jpg

    These wires were run back to a C-Bus relay unit. In this case I have used an L5108RELVP.

    4.jpg

    The URL in the logic code looks at my local weather station. In the attached example code this is pointed to the 'Latest Weather Observations for Maroochydore'.
    This checks the current rainfall rate in the table and stores it in a variable. To set off the chicken the rain rate must be 0 the last time it was checked and greater than 0 on the current check.
    Once rain is detected the relay channel is pulsed on for 1 second and the chicken will begin its new rain notification routine. The routine lasts for about 30 seconds and is audible enough to alert (irritate) all family members.

    Constants
    WeatherURL = 'http://www.bom.gov.au/products/IDQ60901/IDQ60901.94569.shtml';

    Global Variables
    s: array [0..8999] of char;
    tokenPos: integer;
    subs: array[0..999] of char;
    tdCount: integer;
    LatestRainFall: real;
    retryCount: integer;

    Initialisation
    //this must start above 0 so the chicken is not triggered on the first complete cycle
    LatestRainFall := 1;

    Module
    once LatestRainFall > 0 then
    begin
    //enable the chicken
    LogMessage('Trigger the chicken');
    PulseCBusLevel("Local", "Lighting", "The Chicken", 100%, "0s", "0:00:01", 0%);
    end;

    //get the current weather observations for Maroochydore
    GetHTTPData(WeatherURL);
    delay(2);
    retryCount := 0;
    s := '';
    repeat
    ReadHTTPData(s);
    retryCount := retryCount + 1;
    delay(2);
    until (s <> '') or (retryCount >= 3);
    if retryCount >= 3 then
    LogMessage('no data');

    if s <> '' then
    begin
    //to get the value of the rainfall since 9am we need to pull the data apart
    //get the position in the data for the start of the html table data
    tokenPos := pos('<tr class="rowleftcolumn">', s);
    copy(subs, s, tokenPos, length(s));
    //throw out all the unwanted table entries (we want the 13th entry)
    for tdCount := 0 to 12 do
    begin
    tokenPos := pos('<td>', subs);
    copy(subs, subs, tokenpos + 4, length(subs));
    end;
    //remove the remaining unwanted data
    tokenpos := pos('</td>', subs);
    copy(subs, subs, 1, tokenPos - 1);
    //store the extracted value in a local variable
    LatestRainFall := StringToReal(subs);
    LogMessage('Rainfaill: ', LatestRainFall);
    end;

    //check every 10 minutes
    delay(600);
     
    Last edited by a moderator: Aug 19, 2010
    Garfield, Aug 19, 2010
    #4
  5. DoctorZoidberg

    ashleigh Moderator

    Joined:
    Aug 4, 2004
    Messages:
    2,391
    Likes Received:
    24
    Location:
    Adelaide, South Australia
    I want a photo of the chicken from the front!

    Can you record him / her singing about the rain?

    Let me guess, the chicken sings: "Singing in the rain"?!?!?!
     
    ashleigh, Aug 19, 2010
    #5
  6. DoctorZoidberg

    Ingo

    Joined:
    Dec 2, 2006
    Messages:
    290
    Likes Received:
    1
    Location:
    South Africa
    And here I thought I would never use this function if it was available in a PAC. How about getting it ported to a PAC?

    Ingo
     
    Ingo, Aug 19, 2010
    #6
  7. DoctorZoidberg

    Newman

    Joined:
    Aug 3, 2004
    Messages:
    2,203
    Likes Received:
    0
    Location:
    Adelaide, South Australia
    How would a PAC be able to connect to the internet?
     
    Newman, Aug 19, 2010
    #7
  8. DoctorZoidberg

    Darren Senior Member

    Joined:
    Jul 29, 2004
    Messages:
    2,361
    Likes Received:
    0
    Location:
    Adelaide, South Australia
    You could get the PAC to connect to a weather station like Nick has done.

    I realise that this is a terribly low-tech way of finding out if it is raining outside, but sometimes the simplest solutions are best ;)
     
    Darren, Aug 20, 2010
    #8
  9. DoctorZoidberg

    Ingo

    Joined:
    Dec 2, 2006
    Messages:
    290
    Likes Received:
    1
    Location:
    South Africa
    I see what Newman means... the PAC has no Ethernet interface. The only Ethenet connection I have is via my CNI and that won't help. Nick's weather station idea is exactly what I had in mind.

    Ingo
     
    Ingo, Aug 20, 2010
    #9
  10. DoctorZoidberg

    Garfield

    Joined:
    Dec 3, 2004
    Messages:
    24
    Likes Received:
    0
    Location:
    Adelaide
    Garfield, Aug 23, 2010
    #10
  11. DoctorZoidberg

    ashleigh Moderator

    Joined:
    Aug 4, 2004
    Messages:
    2,391
    Likes Received:
    24
    Location:
    Adelaide, South Australia
    Crikey. After the first shower of rain I think I'd want to strangle the hell out of that chicken.
     
    ashleigh, Aug 23, 2010
    #11
  12. DoctorZoidberg

    Ambro

    Joined:
    Nov 23, 2010
    Messages:
    104
    Likes Received:
    1
    Location:
    Adelaide
    This is great; but I have a question:

    I have added this to my project
    Code:
     SetRealSystemIO("Rainfall Measurment", LatestRainFall); 
    after the
    Code:
     LogMessage('Rainfaill: ', LatestRainFall); 
    & set a User System IO called 'Rainfall Measurement'. This is a 'real' system IO.

    I've then added a level indicator component to my CTC's home page; setting the System IO's key function to 'status', System IO variable to 'Rainfall Measurement' (user defined), Visual Properties value tab to 'System IO value Default'.

    After connecting to the network and selecting simulation mode I've waited for the value to change from 0.0. Nothing has occurred.

    So I've open the html page and tried to deduce what Garfield has done. I have VERY limited skills in this area this but it all looks O.K.

    I've monitored the log and it always says the rainfall (rainfaill for the cut & paste ppl like me :p) to be 0. This leads me to believe the code Garfield has been parsing the BOM web page with has changed its layout and broken

    Can someone see something obviously wrong with how I'm trying to display the data or has the parsing of the website changed (without me fully understanding what I'm looking at?)

    Any input greatly appreciated as this is the sort sort of stuff I've wanted to do for quite a while - whilst I won't do the funky chicken I'll parse this and other BOM info to make house projects truely 'smart'

    Also it has been raining alot here in the Adelaide hills, hence a topic of interest which has been on my mind of late :D

    Cheers,
    Ambro
     
    Last edited by a moderator: Jun 26, 2012
    Ambro, Jun 26, 2012
    #12
  13. DoctorZoidberg

    Darren Senior Member

    Joined:
    Jul 29, 2004
    Messages:
    2,361
    Likes Received:
    0
    Location:
    Adelaide, South Australia
    It looks like the data format may have changed slightly. The rainfall data is now the 14th entry in that table, not the 13th.

     
    Darren, Jun 27, 2012
    #13
  14. DoctorZoidberg

    Ambro

    Joined:
    Nov 23, 2010
    Messages:
    104
    Likes Received:
    1
    Location:
    Adelaide
    getting there slowly ....

    so
    Code:
     for tdCount := 0 to 13 do
    begin
    tokenPos := pos('<td>', subs);
    copy(subs, subs, tokenpos + 4, length(subs));
    end;
    points to the <td>

    but it is listed as, for one entry at least :

    <td headers="t1-press-msl">-</td>

    So the question is now : can I 'wildcard' these td's ending in '>' because I assume the previous format was that of <td> with four extra characters (the +4 in the tokenpos)...

    any example would be good but if not I'll keep on trying ... :D
     
    Ambro, Jun 28, 2012
    #14
  15. DoctorZoidberg

    DarylMc

    Joined:
    Mar 24, 2006
    Messages:
    1,308
    Likes Received:
    49
    Location:
    Cleveland, QLD, Australia
    Hi Ambro
    Did you get the warning about the character length of the string?
     
    DarylMc, Jun 28, 2012
    #15
  16. DoctorZoidberg

    Darren Senior Member

    Joined:
    Jul 29, 2004
    Messages:
    2,361
    Likes Received:
    0
    Location:
    Adelaide, South Australia
    The change actually makes the HTML much easier to parse, as each item in the table is now uniquely identified. You no longer need the loop to jump over the first 13 nodes.

    Code:
    ...
    tokenPos := pos('<td headers="t1-rainsince9am">', subs);
    copy(subs, subs, tokenpos + 30, length(subs));
    //remove the remaining unwanted data
    tokenpos := pos('</td>', subs);
    copy(subs, subs, 1, tokenPos - 1);
    ...
     
    Darren, Jun 29, 2012
    #16
  17. DoctorZoidberg

    Ambro

    Joined:
    Nov 23, 2010
    Messages:
    104
    Likes Received:
    1
    Location:
    Adelaide
    Yes - to grab the amount of data I think we need results in a memory overflow.....

    I've set the Global Variables as:

    s: array[0..17999] of char;
    tokenPos: integer;
    subs: array[0..999] of char;

    I've changed s: to a much larger number because I've done a character count to where our data resides and it has given me a total of about 17500 characters without spaces & close to 20000 with spaces.

    If I try making the s: array[0..20000] of char; then I get memory overflow error R104. :(


    Thanks Darren - I obviously was thinking too hard about rearranging the existing code without thinking of the straight forward approach :eek:.

    I believe the code below is good; I still get a 0 rainfall read because I believe the array of characters is now limiting the parsed data.

    One last question: can the unwanted data in the beginning of the BOM site can be skipped over? - for if it can't I think this little black duck (chicken) is cooked .....


    Code:
    //get the current weather observations from the Constants Section
    GetHTTPData(WeatherURL);
    delay(2);
    retryCount := 0;
    s := '';
    repeat
    ReadHTTPData(s);
    retryCount := retryCount + 1;
    delay(5);
    until (s <> '') or (retryCount >= 3);
    if retryCount >= 3 then
    LogMessage('no data');
    
    if s <> '' then
    begin
    //to get the value of the rainfall since 9am we need to parse the data
    //get the position in the data for the rainfall part of the table data
    tokenPos := pos('<td headers="t1-rainsince9am">', s);
    copy(subs, s, tokenpos + 30, length(s));
    
    //remove the remaining unwanted data
    tokenpos := pos('</td>', subs);
    copy(subs, subs, 1, tokenPos - 1);
    
    //store the extracted value in a local variable
    LatestRainFall := StringToReal(subs);
    LogMessage('Rainfall: ', LatestRainFall);
    WriteLn( 'LatestRainFall', LatestRainFall);
    SetRealSystemIO("Rainfall Measurment", LatestRainFall);
    end;
    
    //check every 1 minutes ***change to 10 after finished testing
    delay(60); 
    
     
    Ambro, Jun 29, 2012
    #17
  18. DoctorZoidberg

    DarylMc

    Joined:
    Mar 24, 2006
    Messages:
    1,308
    Likes Received:
    49
    Location:
    Cleveland, QLD, Australia
    In my case I am sure the url was resposible for the warning.
    I found a google link which shortend the url and got rid of that.
    http://goo.gl/

    Thanks for posting your efforts here.

    I saw the original post some time ago but it has taken me until now to slightly begin to understand.

    Hopefully I can follow what you are doing and get it to work too :)
     
    DarylMc, Jun 29, 2012
    #18
  19. DoctorZoidberg

    Darren Senior Member

    Joined:
    Jul 29, 2004
    Messages:
    2,361
    Likes Received:
    0
    Location:
    Adelaide, South Australia
    There is no way to work around this at the moment. The BOM may have other URLs which give the forecast in a more compact form.
     
    Darren, Jun 29, 2012
    #19
  20. DoctorZoidberg

    Ambro

    Joined:
    Nov 23, 2010
    Messages:
    104
    Likes Received:
    1
    Location:
    Adelaide
    You little beauty - just before beer-o-clock and we have a result :D

    Tip for everyone else; try using the 'other formats' link at the BOM site - namely the .json page.

    Thanks all so far, that bevy is going to taste all the more sweeter now this chicken is off my back....
     
    Ambro, Jun 29, 2012
    #20
Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.
Similar Threads
There are no similar threads yet.
Loading...