Weather radar images on 6.4" C-Touch screen pages.

Hi Brad,

Feedback for you from my CTC3, on a 554x480 html component, testing your iframe html pages on the CTC3 hardware:

http://www.noushouse.com.au/CBUS/weathertest.html - cannot see any animated radar images, just other page components from the BOM website page.

http://www.noushouse.com.au/CBUS/congtest.html - Top two elements look OK, bottom one has repeated overlapping text.

 

Hi Brendan,

I have it set at 552 x 480 with left at 88 and it works on the CTC2 as does the weather radar loop as you can see on the photos. Guess I won't be buying a CTC3 ..............Just kidding.

Don't know why the CTC3 would not handle the iFrame the same as the CTC2. Any Ideas?

Brad

 

Hi Brad,

Just to be thorough, I retested your above two iframe html pages on my CTC3 hardware using your precise html component size and location, but still with the same undesired results I reported above.

We may need a CIS Engineer to provide an account for the difference in rendering of iframes between the CTC2 browser and the CTC3 browser.

 

Hi All,

I have added the BOM copyright attribution to the HTML code at weathertest.html. Please note that the attribution is a requirement of using the Bureau of Meteorology copyright image material. See below on CTC2. CTC3 still no go I understand (Thanks Brendan) and issue is being looked into.

Sorry for the below:......................................

If you want to use the BOM images on your own equipment, please seek your own legal advice and approach www.bom.gov.au to ensure your implementation meets the attribution criteria and also whether any licensing fees are required for your use of the material.

Please note the method suggested was for demonstration purposes only to show how an iFrame may be used to display your own data or other data from the web on the CTC. It is not intended to convey any amelioration of your personal compliance requirements to access any copyright material.

All the best,

Brad

 

For those that are interested, below is an animated gif version of sequence of weather radar images that we are able to produce on the version 3 colour C-Touch screen home automation controller:



Being a GIF, the above image is of lower quality than that seen on the C-Touch screen, but it does at least provide an indication of the result produced.

As previously noted, the method we use to produce this requires support from a Windows PC on the same LAN as the CTC with an Internet connection.

This method works fine on the latest version 3 C-Touch screen (CTC3), and is able to run indefinitely without any risk of crashing the CTC3. We expect that it would also would fine on earlier versions of the CTC.

The example shows Sydney rather than Melbourne, because Sydney happened to be experiencing more interesting cloud formations at the time.

 

Hi All,

It is with great excitement (to me at least, as some of you probably had this working 5 years ago!! but chose not to post it........ ... ) that I announce that I wrote a new working CTC logic based program allowing you to see nicely displayed near full screen BOM radar image loops on your CTC, without using the more memory hungry and seemingly CTC3 hating iFrame code model I posted previously. Though both methods I have posted are self supporting within the CTC and don't need an external PC or other device to serve up the radar image loop (other than the normal BOM website itself), the new method is far more efficient.

This completely new program uses the radar image files directly and looks the same as my previous screen photo (or Brendan's nice loop screenshot - love those image buttons!!) except that I now fully retain the original copyright attribution. The code has been running for several 1-2 hour stints over the weekend in my CTC2 while I've been finessing it and seems stable and not overtly memory hungry.

The code looks a bit like a dog's breakfast at the moment with comment lines and un-optimised code etc so I will spend a little while cleaning it up, documenting the method and making it presentable before handing it over. I will also let it run for a full 24 hours to ensure there's no time/date parsing nasties in there.

So I should have the code to post back here fairly soon for all to enjoy. Of course, if you or your customer has a PC on the same network, Brendan's code will offload the fairly solid processing overhead from your CTC which could work better should your CTC be doing lots of other things.

So you now have a choice of 4 methods for display of BOM radar images (3 loops/static and one static only). Choose the one that best suits your CTC installation.

Be back real soon with the code after some final testing, tidying and documenting.

Brad


PS. Anyone who wants a warts and all pre-release to muck around with, please PM me. Brendan, I will be emailing you a copy for testing on your CTC3, Thanks if you can test.

 

Hi All,

I have been awaiting a call back from the BOM copyright officer (for a few days) to ensure that any image based method (ie bypassing BOMs direct public page and using the images only) fully complies with their copyright policy. On my reading of the copyright policy, the method seems to comply, but proper due diligence requires getting it confirmed before posting on an open forum. As soon as I have confirmation it's OK, my method will be posted.

Further testing has seen it run for several 24hr sessions with minimal memory issues. There's a small bleed of ~1-2MB over 24 hours, but I think that is probably reasonably normal for most browsers and it is only if the CTC is left on weather radar 100% of the time which is probably unlikely anyway.

I have also been working on a tidied up new version that uses the same method keeping up to 6 loop pages (different KMs, wind, rain etc) updated so when your image button goes to the page, it also has the latest information there. It spreads out the processing to minimise interference with other functions you might have operating. As usual, the main limiting factor is really memory and lots of image based pages chews it up fairly quickly.

Again, the method will be posted once I hear back from the BOM but I am happy to give an interim 'method demo/testing' version to anyone who would like to beta test it, on the basis you accept the copyright compliance responsibilities personally, it is for personal use only, is not given to a client or posted elsewhere etc.

All the best,

Brad

 

Here's some icons for those interested. You'll need to resize them - they are big but at least you can scale them to any size you want

 

Hi 7iron,

Those are some very nice looking icons and thanks for posting them.

Also just taking the chance to pass on to those interested, an update regarding the BOM copyright arrangements for radar loop images and the background transparencies.

After speaking at length with the copyright officer at BOM yesterday, it became apparent that any automated full time scraping of images from BOM's main production website by an external system constitutes 'use of material in a way that is beyond the scope of the terms of use'. To legally access the images, this requires the user/installer to complete (and have accepted/returned by BOM) a Bureau Access Agreement - see sample.

Though you might consider this an unnecessary step, I need to pass on the 'stick'. The officer told me that the BOM is very concerned about any use that would potentially jeopardise BOMs ability to provide weather information to normal public web users. If everyone was scraping full-time with various devices, it could lead to the public facing server being overwhelmed during major weather events. You will note that the normal BOM radar loop web page times out after a certain period.

I was told that any IP address that hits BOM continually on their primary site will, after a fairly short period, be automatically blacklisted and should it continue if it gets reconnected, will be put into a permanent ban pool. As most people have a single 'outside' IP address for their broadband, this would effect all BOM site access from your/your client's site. Based on this, the copyright officer strongly recommended (and I concur) everyone who wants to use the image loops fill in the Bureau Access Agreement, and access the images at an alternate FTP site that is provided for those who have a valid Bureau Access Agreement.

I will post the method to access the nicely centered, non-scrolled single loop from BOM updated solely by the CTC, and also a multi-page radar loop (eg rain, wind, 64k, 128k, national etc) when both have been updated to use the FTP location once I receive my B.A.A. (I think will possibly be some time tomorrow).

I have re-written the scripting and the method now does not bleed memory (well minimal - almost none). For example, I have had the radar loop running full-time for 6 days now without issue or reboot.

So that is where it is currently with any image based BOM loop method.

Back Soon,

Brad

 

Australian BOM Radar Loop - Method to nicely display on CTC

Hi All,

Here is the method to create a single nicely centred non-scrolling radar loop on your CTC that is wholly created by the CTC. If you have a PC handy, you can use Brendan Roger's offered method which will create less load and memory use in your CTC. This is a stand-alone method using CTC resources.

I call this version CTC_BOM-RL_5 v1.01 and the new version is CTC_BOM-RL_6. The reason for the long winded title is that I also have new CTC6 v 1.0 logic module almost ready (in final trialling) which keeps all the radars updated, ie all ranges, wind and rain radars. After a bit more testing I will be posting the up to 6 loop code as I am currently optimising the memory use and modifying for the BOM FTP site rather than the primary site.

This version is quite basic compared to v6 but is probably best for someone who just want the main radar page for their area, for like a rain/no rain sort of analysis (take the umbrella today? at the front door perhaps). It is very reasonable with memory use.

How it works:

There are several elements to the method.

1. A algorithmic radar image UTC time key generator - Timewatch module (previously generated using a page parser of the BOM site)

2. A Local HTML page generator (MakeRadarFile module) that using procedures creates an HTML 'sandwich' made up of:
- a Prefix HTML file - (the first half of the web page)
- The output file which has the current radar loop image URLs generated by element 1.
example [noparse]
radarimages("ftp://ftp.bom.gov.au/anon/gen/radar/IDR713.T.201307111030.png","ftp://ftp.bom.gov.au/anon/gen/radar/IDR713.T.201307111036.png","ftp://ftp.bom.gov.au/anon/gen/radar/IDR713.T.201307111042.png","ftp://ftp.bom.gov.au/anon/gen/radar/IDR713.T.201307111048.png","ftp://ftp.bom.gov.au/anon/gen/radar/IDR713.T.201307111054.png","ftp://ftp.bom.gov.au/anon/gen/radar/IDR713.T.201307111100.png")[/noparse]

- a Suffix HTML file to finish the web page (ie second half of web page)
It uses the ReadPrefix and ReadSuffix procedures to import the HTML parts before re-assembling the output HTML page 'weatherradar.html'

3. Coded into the HTML page is a small javascript slideshow that makes the image loop run

4. An HTML element is inserted into a page on your CTC PICED project. You an adjust the size of the Window using the HTML element size and also in Prefix.html

Every Six Minutes, the Timewatch procedure runs to create a new local 'weatherradar.html' file that contains the very latest number of images which are settable between 11 and 3 (I have made 6 the default, 1 more than BOM ie the last 30 minutes). It updates the file when the BOM image delay timer elapses ensuring you catch the very latest images as soon as they are posted.

No images are actually transferred unless you are in the page with the HTML element, just an HTML page is built in the background READY to LOAD. This new version does not continually hit the BOM website to parse the image names, nor download anything until you have the actual page open.

The whole complete web page file is weatherradar.html. See parts listed as Prefix.html, Suffix.html and Outputfile.html. I have had to put these files on my website as the forum system reads posted HTML and Javascript and runs it (??) even with the bb code noparse and code tags surrounding it. (If someone has a clue how to directly post an example HTML file on the forum, please let me know and I will update post.)

To make this run, follow this procedure:

I am calling it CTC_BOM-RL_5 v1.01. This new version you can leave on radar continuously with almost zero memory leakage. Of course this will not be usual operation.

To be very memory frugal, you do however need to create your own background file for your area's radar, which is just a merged PNG of all the elements from the radar that you want, the usually tick boxed items on the BOM page. These are available here.


As I can't upload HTML files here, I have put the latest files into a directory on our server in www.noushouse.com.au/CTC5 directory

In your browser, just highlight each file and do a right mouse button 'Save Link As' into your PICED project directory.

files that you will transfer to your CTC are:
1. IDR713.backgroundmerged.png (this is the Sydney one - start with getting this working, then change to your zone etc)
2. PrefixFile.html
3. SuffixFile.html
4. weatherradar.html (what the final file it creates every 6 minutes ends up like, put this into your on page HTML element in PICED - initially size to 540 x 480 you can change it later to whatever you want)

other support files:
5. Logic Summary.htm - is the logic modules report from PICED, containing the 2 logic modules, 2 procedures, initialisation and Global variables you need to add to your PICED project.
6. CTC5_version.txt - just a file with the latest version number and log of issues and fixes.

To start off, just make a project with a page with a HTML element pointed at C:\weatherradar.html. It should load OK in simulation mode. Create all the logic elements (modules, procedures, initialisation and global variables) noted in the Logic Summary file.
When you upload the project to a CTC screen you just need to also set it to upload the 4 files.


Current Issues:
1. Had to change 'For Do' loops to 'While Dos' as the logic engine seemed to ignore the For Do after project became a certain size for reasons unknown and currently undiagnosed. Put in While Dos, but that just made it use slightly more processor and have much messier code. Will fix in later version.

2. Background image currently needs to be separately combined by downloading topography, locations and background (as z-index, CSS and other normal mechanisms aren't working to overlay transparencies and chew up too much extra RAM anyway). I just used Illustrator but Paint Shop etc is fine.

3. Overlay image of radar range - still to update


I would like to hand this method over to the community and welcome suggestions of code changes and enhancements and I will incorporate them and release these acknowledged in the CTC5_version.txt file.

The method runs well in PICED in simulation mode so test it and get working in there before you transfer to CTC. Look in the log for program cues being logged. Additional testing log entry lines are built into the code. remove '//' comment prefixes from lines to log that lines output to the log file for testing purposes if required.

To use this method, as discussed in the previous post, you will need to sign a Bureau Access Agreement, a fairly simple procedure you can start emailing Bureau's copyright officer: [email protected]. At this time, all users must apply for a BOM bureau access agreement for continued use to comply with copyright requirements of BOM. Please be aware you need to accept personal responsibility for ensuring you and your client's copyright compliance. Downloading files and use of the method constitutes your agreement.

Also, any issues, questions, critiques, code suggestions, please PM me. There is plenty of information in comments within the logic and HTML files.

Enjoy & I hope it works well for you.

Brad

 

Here's some icons for those interested. You'll need to resize them - they are big but at least you can scale them to any size you want

 

Hi Brad,

I had a chance to test your CTC radar loop code overnight and today. You have obviously put a lot of time and effort into this, and the way in which you are trying to achieve this is appealing.

Test results after being transferred to a Version 3 CTC:

Initially worked fine.

I then had no problem changing the location/range and background images to Melbourne, and it continued working fine.

However, over a 24 hour period the number of recent radar loop images being successfully displayed on the CTC has been diminishing image-by-image down to there now being no current radar images being displayed in the loop.

I set the viewing page to time out after 60 seconds and go to another page that does not display the radar loops, and was periodically going to the V3 CTC to manually check the results.

I hope the above feedback is helpful.

 

Hi Brendan,

Thanks for your comments. Yes it was a bit of work to minimise the memory usage with the ultra minimalistic javascript slideshow and replacing a standard page parser which used a lot of memory and continually hit the BOM site, (both bad things) with a algorithmic image file name generator that self calculates the UTC time based file names.

Of course, my CTC2 is 'smiling at me' with its perpetually open BOM loop window that has been running constantly for 9 days and only got updated to that new code the day or so before I posted it. Since then, it's been running for 2.5 days and I haven't seen the loop miss an image.

I will email you directly to diagnose a bit more, but it will be in the image file name generator 'TimeWatch' module. Possibly, the FTP file name on line 172.

If you get a chance, can you run it in PICED for about 30-45 minutes and then send me the LOG file from PICED. It will show you the files it is trying to hit and from that you can try them in your normal browser as they should work directly.

Thanks for trying it and I am sure we can get it fixed quickly,

Brad

 

Hi Brad,

A quick update.

The number is radar loop images being displayed has steadily increased again since my above post, first displaying one image, then two, three, and now four. Apparently a sync issue.

 

Hi Brendan,

It's the self healing algorithm kicking into action

Based on the timing, the logic is not crossing any time 'edges' such as day change etc, and it does the hourly change every time without incident.

That reminds me, I need to add an 'Issue' that I had forgotten, that being at month change as I didn't bother filling it up with all the additional logic for month and year changes. Suffice to say, at month change and year change, it will stop for a few hours as that logic is not there. I can handle that, but if you don't like it, add another if/then under the hour and day changer lines.

Can you try something for me? First, set the HTML update time back to 120 seconds. The page only gets refreshed every six minutes so re-loading the HTML page every minute actually chews up more memory than it needs to.

Second, in initialisation, can you change the radar delay from 5 to 6 minutes. I think this will fix your problem. BOM posts its images about 5 minutes late to the main site, but it may take longer on the FTP site. As noted before, run the logic and PICED, and it should perfectly replicate the loop. If it has the problem, look at the Log file and see the 6 images it is adding to the output file. If it is the delay being slightly out, the last image (that being the current one) will give a 404 in a browser until the image actually arrives.

If the above doesn't resolve, as per previous post, please send your LOG file. You can turn on some of the additional LOG lines if you think it will help show the issue.

Does it look good on your CTC3? No issues like the iFrame?

Thanks,

Brad

 

Hi Brad,

Your page is looks quite ok now on the V3 CTC, it looks similar to my static radar page at the top of the first page of this thread, except that your border seems to be a little uneven (widest at top, and RHS wider than LHS) and all appear to be wider than is necessary (all of your margins are wider than those on my static page). Also, you have disabled the scroll bar, which might also be better enabled in your case as well.

Which HTML update time are you referring to? (I have the CTC radar loop viewing page auto-refresh disabled at the moment).

Radar_Delayed changed from 5 to 6 min.

Will let you know how it goes.

 

One other thing Brad,

On the BOM radar loops the only layer behind the weather radar image seems to be the terrain layer, whereas the map, range, and locations layers seem to be above the radar layer.

 

Hi Brad,

So far so good, not loosing any radar image frames now on the V3 CTC.

By the way, I have tried reducing the delay between frames below your initial value of 1200ms, and it seems to be working fine all the way down to 50ms on my screen. 100ms is probably as low as you would want to go.

At higher frame rates, it would be nice to be able to support a larger number of frames, say 10-20.

 

Hi All,

Thanks for the great feedback. I have pulled back some features I had already put into CTC6 to make CTC5 better as many of Brendan's questions/issue had already been addressed in CTC6.


v1.02
Whats New:
1. Adjusted timers in SuffixFile.html to make image movement more fluid.
2. Added #radarbackground and #radarforeground CSS styles to allow images of range, topography, location to be easily added in PrefixHTML body. Users can add any transparencies they want into the #radarxxxxground DIVs. I suggest that you combine multiple overlays into a single foreground and background file to reduce memory usage. (fix of V1.01 #2 & #3 issue)
3. Changed positioning from fixed to absolute to combat IE positioning bug to allow #2 to work
4. Adjusted some window sizes in Prefix to make output look more like standard BOM radar image (to address feedback from Brendan Rogers)

Notes:
1. If you want a nice looking image without the timestamps - use width 496 height 496 in the #radarbackground, #radarforeground & #radargallery img CSS.
2. If you want a full resized duplicate of BOMs radar page with Timestamp - use width 470 height 470 in the #radarbackground, #radarforeground & #radargallery img CSS.
3. For #1 and #2 sizing, set HTML element size to 480x480(#1) or 496x480(#2)
4. If you are using individual transparencies for location, topography, range etc, don't forget to upload them to CTC when you do your transfer.

Current Issues:

1. need tweak to algorithm slightly as it created - [noparse]ftp://ftp.bom.gov.au/anon/gen/radar/IDR713.T.201307132400.png[/noparse] - hour needs fix
2. v1.01 issues #1


The Files:
Logic Summary.htm
IDR713.range.png
IDR713.locations.png
IDR713.topography.png
IDR713.background.png
PrefixFile.html
CTC5_version.txt
SuffixFile.html
weatherradar.html
I have updated the files to CTC5-V1.02:

In your browser, just highlight each file and do a right mouse button 'Save Link As' into your PICED project directory.

files that you will transfer to your CTC are:
1. PrefixFile.html
2. SuffixFile.html
3. weatherradar.html (what the final file it creates every 6 minutes ends up like, put this into your on page HTML element in PICED - initially size to 496w x 480h you can change it later to whatever you want)
4. IDR713.background.png (this is the Sydney one - start with getting this working, then change to your zone etc)
5. IDR713.topography.png (this is the Sydney one - start with getting this working, then change to your zone etc)
6. IDR713.locations.png (this is the Sydney one - start with getting this working, then change to your zone etc)
7. IDR713.range.png (this is the Sydney one - start with getting this working, then change to your zone etc)

other support files:
8. Logic Summary.htm - is the logic modules report from PICED, containing the 2 logic modules, 2 procedures, initialisation and Global variables you need to add to your PICED project. LOGIC IS UNCHANGED FOR v1.02 ie same as v1.01
9. CTC5_version.txt - just a file with the latest version number and log of issues, notes and fixes. PLEASE READ - UPDATED


Brendan, you are correct, you can make it change as fast as you like. The Javascript window changer is micro and so also very fast. You can also update the radarindex:=6; variable in initialisation to 11 (current tested max) to give you 11 images in the loop. Try it with more and see if it breaks!! I think it will as current logic can only go past one hourly boundary in UTC calc.

In answering your comment regarding sizing and positioning, I have added further notes to PrefixFile.html to give installers/users more information about sizing, but really I see sizing and positioning of the file image loop as being the domain of you, the installer to adjust it to properly fit within your template running and any bespoke menu buttons your are using.

Anyway, hope that update addresses most of the issues. There is a small 7 image issue at UTC time day change that I am yet to test the resolution of. I don't want to send out modified code until it is tested. Logic remains unchanged from v1.01 so you don't need to change anything there.

Brendan, thanks for confirming it is working well on your CTC3 (Phew! - that's a relief). I have it running nicely on the CTC2

To get you guys thinking of further uses of the 'Stuff the HTML sandwich' technique, you could easily write a sequential number incrementer to replace 'Timewatch' and have it pull down images from a private server for a picture slideshow eg. IMG_9001,IMG_9002,IMG_9003 etc.

To use this method, as discussed in the initial method post, you will need to sign a Bureau Access Agreement, a fairly simple procedure you can start emailing Bureau's copyright officer: [email protected]. At this time, all users must apply for a BOM bureau access agreement for continued use to comply with copyright requirements of BOM. Please be aware you need to accept personal responsibility for ensuring you and your client's copyright compliance. Downloading files and use of the method constitutes your agreement.

Good Luck with the update. Any feedback, please PM or email me,

Brad

 

Hi Brad,

Presentation on V3 CTC is much better now, with no margin on left or top.

Tried with radarindex=11, it was sometimes showing all 11 frames, and sometimes showing a lesser number.

Set radarindex back to 6, and at the moment it is only showing one or two frames correctly.

Will contact you about CTC log file contents.