Http Data

Discussion in 'C-Touch/HomeGate/SchedulePlus/PICED Software' started by DoctorZoidberg, Jun 13, 2012.

  1. DoctorZoidberg

    DoctorZoidberg

    Joined:
    Aug 8, 2010
    Messages:
    12
    Likes Received:
    0
    Location:
    The Year 3000
    Hi,

    Been playing with GetHttpData some more.
    Have a web page that has approx 40000 characters. I am interested in characters around the 20000 mark . Using ReadHttpData into array how can I get the characters Im interested in. Obviously cant make array that large as will be memory overflow error.

    I have this working on smaller pages (around 3000 characters) I read them all into a array (0..3000) then get data as I need.
     
    DoctorZoidberg, Jun 13, 2012
    #1
  2. DoctorZoidberg

    Darren Senior Member

    Joined:
    Jul 29, 2004
    Messages:
    2,361
    Likes Received:
    0
    Location:
    Adelaide, South Australia
    At the moment there is no way easy around this.

    You could try to read the HTTP data in bits using a regular socket, but it is unlikely to make any difference.

    There are plans to increase the stack size of the logic engine, but we weren't thinking of doing it soon (issue number 22706). If there is enough interest in changing this, we will take that into consideration when looking at priorities.
     
    Darren, Jun 14, 2012
    #2
  3. DoctorZoidberg

    ssaunders

    Joined:
    Dec 17, 2008
    Messages:
    240
    Likes Received:
    35
    Location:
    Melbourne
    +1 on increasing the stack/heap or both sooner rather than later, Darren. Every day that goes by we see bigger and bigger web pages that could be handled that must be put somewhere in entirity.

    My most recent use case thankfully has a work-around. I'm pulling weather data from a feed provider, and can specify in URI parameters the data that I want. Forecast, observations, warnings, etc. I split the reads to pull the different data types, even though they're actually coming from the one URL.
    There are three issues:
    1. The code would be simpler if it could be a single read
    2. I can't get all the data I want
    3. Variable sized responses could be better accommodated

    Issue 3 is a pretty big one. For example, if I'm reading weather warning information I can't determine how many warning or cancellation records I am going to receive. Each warning entry in the XML returned is pretty big, and if I get more than two or three then bad stuff happens...

    Cheers,
    Steve
     
    ssaunders, Jun 1, 2013
    #3
Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.