Http Data

Hi,

Been playing with GetHttpData some more.
Have a web page that has approx 40000 characters. I am interested in characters around the 20000 mark . Using ReadHttpData into array how can I get the characters Im interested in. Obviously cant make array that large as will be memory overflow error.

I have this working on smaller pages (around 3000 characters) I read them all into a array (0..3000) then get data as I need.

 

At the moment there is no way easy around this.

You could try to read the HTTP data in bits using a regular socket, but it is unlikely to make any difference.

There are plans to increase the stack size of the logic engine, but we weren't thinking of doing it soon (issue number 22706). If there is enough interest in changing this, we will take that into consideration when looking at priorities.

 

+1 on increasing the stack/heap or both sooner rather than later, Darren. Every day that goes by we see bigger and bigger web pages that could be handled that must be put somewhere in entirity.

My most recent use case thankfully has a work-around. I'm pulling weather data from a feed provider, and can specify in URI parameters the data that I want. Forecast, observations, warnings, etc. I split the reads to pull the different data types, even though they're actually coming from the one URL.
There are three issues:
1. The code would be simpler if it could be a single read
2. I can't get all the data I want
3. Variable sized responses could be better accommodated

Issue 3 is a pretty big one. For example, if I'm reading weather warning information I can't determine how many warning or cancellation records I am going to receive. Each warning entry in the XML returned is pretty big, and if I get more than two or three then bad stuff happens...

Cheers,
Steve