Category: Blog

My NEXRAD JavaScript libraries have been almost 15 years in the making. And they didn’t start with me plotting my own images.

In the early 2000s websites didn’t have rich, interactive interfaces with dynamically loaded data. Most people point to Gmail as the beginning of these type of AJAX web sites, as they were called at the time back in 2004. Although other web sites did dynamic loading and played other tricks with JavaScript and iframes to achieve similar interfaces.

Weather websites at the time typically had a static image showing the radar, or maybe an animated GIF. This was great. You could see the current radar without having to wait for the 5pm news or for 8 minutes after on The Weather Channel. But for someone who was interested in weather this was still lacking.

Screen Scraping

My first foray into this was through a technique called screen scraping. On my web server I would run a PHP script every few minutes that would check my two favorite radar images at the time: Accuweather and Wunderground. That script would determine if a new image was present and store it to my server. It would keep a total of 5 or 10 images expiring the oldest when a new one arrived.

An example image from https://wunderground.com that would be screen scraped – their NEXRAD service shut down in March of 2022

The web page associated with it would let you view either set of images as an animation by changing out the src attribute on an image tag. This method had it’s drawbacks that have all been addressed in subsequent updates.

Some drawbacks were that images weren’t pre-loaded. On a dial up modem, not uncommon at the time, it might take 3 or 4 times through the loop before all of the images loaded.

Improving the interface

The first two changes implemented allowed for changing the frame rate and and manual forward/back control of the images. Specifically, forward and backward control was accomplished by using the mouse’s scroll wheel. Later when touch screens became prevalent, dragging across the image was added as another way to control this.

This easy manual control via the scroll wheel is something that I have always felt was lacking, and is still, in any radar viewer that I’ve used. For me, this control is critical as an amateur weather enthusiast to be able to follow the storm’s track and watch it’s development.

Excerpt from scroll wheel handler

Next I began pre-loading images. Initially this was done inside of a hidden div, although the method has changed over time. This meant that, especially on a slower connection, that the second and later frames might already be loaded when the animation reached them.

This interface at this point was nice, but if you left it open for several minutes you’d have to manually refresh to see if there were any updates. My initial fix for this was to have JavaScript automatically reload the page every few minutes. But then it got a lot better.

AJAX

In 2009 or 2010 I began learning about Asynchronous Javascript and XML (AJAX). This allowed me to make several changes to the frontend, and supporting changes to the backend. I was now able to load just a “shell” of a page and use an AJAX call to get a list of images from the server to load. This made it very easy to check for a new list every minute without reloading the page.

It also allowed for switching between the Wunderground and Accuweather image sets without reloading the page. And as the image sets grew to include TDWR, satellite, velocity and other regional and national images this became very helpful.

It was around this time as well that I began time stamping images. Or more specifically I would log metadata about the images to a database that included the time I downloaded the image. This was a reasonable estimate of the radar’s time. But it began to make a problem clear. Several of my image sets would occasionally get out of order. I never did sort out exactly why, and it didn’t happen on the source web sites. My suspicion was issues with caching at CDNs or connecting to a different CDN server on subsequent calls that was slightly out of date. This order-of-images problem lived on for almost 10 years.

Database excerpt

Jquery

The backend at this time continued to run on PHP. But around 2015 I started experimenting with Jquery and found that it was a much better way to handle all of the interactivity that I had added to the web page. I recall having so much code specific to Firefox, Chrome and at one point even IE and Jquery did away with this. It also integrated some really simple and helpful transition animations that very much helped show the process flow through the web site.

Jquery was a very nice tool at the time to deal with different browsers but by 2021 it was just about unnecessary. The remaining major players in the browser world had all begun following standards to a much better extent than in 2005 when I started this project. So at this time I refactored the code in two ways: removing Jquery in favor of new standard javascript methods and modularizing the code for easier maintainability.

Example of code that was Jquery replaced by plain Javascript

My Own Images

In 2018, while I was out of town for work and browsing the web aimlessly while stuck in a hotel room I came across a Javascript tool for plotting NEXRAD data. It worked, but it was incomplete with regards to modern compression methods used in the current NEXRAD network. I then set out to figure out where to get radar data and just happened to find that AWS S3 was now hosting both archival NEXRAD data as well as “live” data.

The journey here took a lot of effort. I’ll list some of the problems that had to be overcome:

  • The bzip format used in current radar products was not standard. There were headers interspersed with chunks of bzip data. The headers were not well defined but I eventually was able to sort them out and wrangle a nodejs bzip library to work on this data.
  • The data could be presented in an older format or the new dual-pol/hi-res format. These needed separate handling at the appropriate places to parse the data successfully. Separately, the plotting tool needed to understand the two types to be able to correctly show the data.
  • Plotting was slow, initially. A lot of work was done to speed up the plotting process. On a day with some storms, plotting went from 10s originally down to about 2s.
  • “Live” data, called chunks, took extra effort to process. A single image may come in as 3 or 6 chunks. I had to study a lot of raw radar data to determine whether to trigger on the 3rd or 6th chunk, and if there was another lowest-elevation scan at some other chunk.

I enjoyed the challenge of making all of this work and it solved the problem of out-of-order images that the screen-scraping method occasionally produced.

You can read more about the NEXRAD tools for JavaScript

Serverless

After I had updated the libraries to deal with the new data formats I had to find a way to run these. Previously I had used PHP on my web server that also hosted wordpress. But the new libraries ran on nodejs. This is when I discovered Lambda and it’s ability to run a one-off function with code that I supplied.

Lambda function configuration

So I set about implementing the library in a way that a lambda function could handle, and it was very capable of plotting this data and storing it to S3. Then other metadata and tasks were adapted to the Lambda environment. The final stack looks something like this:

  • An S3 triggered Lambda that reads the first chunk in the set of chunks as they arrive and determines what chunks should be decoded. When those other chunks arrive it then triggers the next process.
  • A plot worker receives the “interesting” chunk numbers from the previous function, downloads all of the data for each chunk, plots it and stores it to S3. The plotting consists of reflectivity, velocity at various resolutions and crops.
  • The plot worker sends a command to a third lambda function that writes metadata to the database for retrieval by the fronted.
  • A separate backend tool that monitors the api.weather.gov for current severe thunderstorm and tornado warnings and stores these to a database in a format that is practical for displaying on the radar map.
  • A frontend “list” api that can return a list of current images for a selected radar site, and that can list all current warnings for the current site.

The frontend queries the API for the list of images and warnings, loads the images from S3 and draws the warnings on the image.

Maps

Radar images by themselves are not hugely helpful. You need some context to view them within. Typically road, state and county maps are shown over the radar plot. I used Open Street Map to provide major highways, county and state borders as a base-image for my radar plots. I downloaded the relevant data and used my own nodejs tool to draw these base maps from the data.

Map created through Node.js with OpenStreetMap data

Conclusion

I hope you have enjoyed this journey and would like to see the resulting web site. However I am keeping that site private. There are two reasons for this. One historical and one is cost. Historically, during the screen-scraping days I am sure there were copyright issues with me copying and re-displaying radar images from other web sites. So I kept the site private.

Today, the images are not copyrighted. In fact, any data produced by an agency of the United States Government can not be copyrighted. But there is now a non-zero cost with producing the images. For me to plot the handful of radars around my location, it’s reasonable. But if I were to open the site to the public I would need to plot every radar site. And I’ve calculated this at $150-200 per month which is well outside of what I could pay for out of pocket.

Tornadoes pass through the Chicago area on June 20, 2021 as displayed by NEXRAD library and frontend described in this article

It’s unfortunate, but I’m still very happy with my own results. And I do give my own web site a lot of traffic especially on days when there’s significant weather in my area.

In addition to the free forecast and observation data that powers Temperature.Express, the National Weather Service also makes available live and archive data from their NEXRAD weather radar network. This data dates back to 1991. I have created npm packages for parsing and plotting this data. [Jump to packages]

Level 2 Data

Level 2 data is what you’re used to seeing in your weather or radar app. It’s the “how hard is it raining” graphics typically in green, yellow and red with red being the heaviest or most severe precipitation. More advanced users will also recognize the red and green velocity plot that shows wind direction and intensity.

In 2019 I found nexrad-radar-data on Github. It worked, but needed some updating to to modernize the code and deal with the new compressed formats that were being stored to the archive. I then added plotting capabilities in a separate package. The results can be seen below.

Example reflectivity output from nexrad-level-2-plot
Reflectivity plot of hurricane Harvey on August 25, 2017
Example velocity output from nexrad-level-2-plot
Velocity plot of hurricane Harvey on August 25, 2017

A commercial tool exists for viewing this data, as does a set of libraries for Python. Neither of these options lent themselves to the environment where I was looking to deploy them. So I went ahead and began improving what was available to suit my needs.

nexrad-level-2-data is the direct descendant of nexrad-radar-data mentioned above and outputs a JSON object containing all of the data from one level 2 scan. An excerpt from this data is below. Additional information on how to make sense of this information including links to formal specifications from NWS are available in the Github repo.

{
  "elevation_angle": 0.4833984375,
  "channel_config": 2,
  "waveform_type": 1,
  "super_res_control": {
    "super_res": {
      "halfDegreeAzimuth": true,
      "quarterKm": true,
      "300km": false
    },
    "dual_pol": {
      "300km": true
    }
  },
  "surv_prf_number": 1,
  "surv_prf_pulse": 15,
  "azimuth_rate": 21.148681640625,
  "ref_threshold": 16,
  "vel_threshold": 16,
  "sw_threshold": 16,
  "diff_ref_threshold": 16,
  "diff_ph_threshold": 16,
  "cor_coeff_threshold": 16,
  "edge_angle_s1": 0,
  "prf_num_s1": 0,
  "prf_pulse_s1": 0,
  "supplemental_data": {
    "sails_cut": false,
    "sails_sequence": 0,
    "mrle_cut": false,
    "mrle_sequence": 0,
    "mpda_cut": false,
    "base_tilt_cut": false
  },
  "edge_angle_s2": 0,
  "prf_num_s2": 0,
  "prf_pulse_s2": 0,
  "ebc_angle": 0,
  "edge_angle_s3": 0,
  "prf_num_s3": 0,
  "prf_pulse_s3": 0,
  "reserved": 0
}

The changes were to the source repo were substantial. As I learned more about the newer radar formats it was necessary to go back and rework the core of the parser so that different message and compression types could be detected at the appropriate places. The data format is well documented but was designed in the 1980’s so there are a lot of less commonly used data structures compared to a modern system. The documentation makes significant use of “half-words” (16 bit values), and has a special floating point representation designed around the exact needs of the radar’s output. Some of the significant changes include:

  • Looking for compression headers and passing data to a gzip decoder. Several gzip blocks might be present within one file and the parser detects these additional blocks and decodes them accordingly. This is not a standard gzip file format.
  • In the mid 2010’s radar sites were upgraded with super resolution which more or less quadrupled the resolution of the data. This change puts all new data formats into the data files and the correct format needs to be determined before decoding can take place. The plotter also needs to identify the two types to scale the resulting images properly.
  • When using live data, it is delivered as “chunks” that are typically 1/6th of a complete scan per file. Various methods were developed to combine multiple independent files and recombine it with header data that only appears in the first file in the series.
  • The data coming from the radar is not perfect. After running the parser and plotter in a production environment for a year, several errors would repeat themselves. Careful analysis of the problematic files with a hex editor revealed truncated files, incorrect length pointers and missing slices of the scan. Best-effort methods were added to try and find the correct lengths, pointers and to flag missing slices.

Level 3 Data

Level 3 data is the result of compiling data across several minutes or hours, or text output of analysis algorithms that flag hail, rotation and other phenomenon. These different types of output are called products.

This data takes on a completely different format from level 2, and for several of the products does not even produce data that can be plotted. Below is the output of Digital Total Accumulation which acts as a digital rain gauge, estimating the amount of rain that has fallen during a thunderstorm event at any point within the radar’s area.

Example DTA output from nexrad-level-3-data
Digital Total Accumulation output from storms passing through the Chicago area on July 23, 2022

Even with the significantly different data format. A lot of techniques from the level 2 software could be leveraged to make decoding level 3 data a simpler task. Concepts like half-words and special floating point and fixed-point binary formats continued to exist.

Other data that is stored in the level 3 format is text-only and may be presented as the pre-formatted text shown below, or in a more structured form that lends itself to a JSON output.

                                STORM STRUCTURE
     RADAR ID 552   DATE/TIME 04:19:21/19:02:24   NUMBER OF STORM CELLS  24

   STORM      AZRAN      BASE     TOP    CELL BASED VIL    MAX REF    HEIGHT
     ID      DEG/NM       KFT     KFT       KG/M**2          DBZ        KFT

     D6      120/ 63    < 6.5    19.6          29             57       15.1
     P2      165/ 83    < 9.7    21.6          22             54       12.6
     T9      190/ 94    <11.7    19.9          21             53       11.7
     U3       78/100    <12.8    27.3          21             52       12.8
     H8      109/159    <27.5    33.5          20             49       27.5
     T6      125/138    <21.5    34.6          19             49       21.5
     C2      114/ 97    <12.2    21.0          19             52       12.2
     K8      201/133    <20.4    25.1          18             50       20.4
     L9      152/ 79    < 9.0    20.1          15             50        9.0
     X0      194/106    <14.1    23.3          14             50       14.1

Level 1 Data

What about level 1 data? Level 1 data is only published as part of select case studies and not on a continual basis like level 2 and level 3. It is the raw data that comes back from the radar antenna in analog form. Upon receipt, it is processed immediately at the radar site into the digital level 2 format that is distributed through the National Weather Service’s various channels.

Acquiring Data

The data is available freely through different cloud providers. A good starting point is Nexrad on AWS which at the time of this writing appears to have the most complete level 2 archives, live level 2 chunks and level 3 data. This AWS source also provides notifications for new data in some of the buckets.

Summary

The complete suite of four npm packages consists of:

Back in the 90s it took a little more effort and planning to get a weather forecast. You could look at the newspaper when it arrived. There was the local morning news. A few radio stations might have it at 10 and 40 after. But for me I would go to Local on the 8s on The Weather Channel. I’ve recreated this with WeatherStar 4000+ and you don’t have to wait until the 8s any more! Oh, and the forecasts are current, no reruns from the 90s.

WeatherStar 4000+ Screenshot

The technology was really impressive at the time. The Weather Channel was on cable and at that time talked about weather forecasts on a national scope. But on the 8’s a local forecast was inserted into what was otherwise a regular, nationally broadcast cable channel. Wow! And if you went on vacation and watched The Weather Channel it was a different local forecast for where you were.

There is a lot of nostalgia for these forecasts. First there’s TWC Classics which documents and discusses the history of The Weather Channel through screenshots and video clips. Then there’s the WS4000 Simulator which strives for accuracy in both look and information that is presented. It runs on desktop OSes.

This project is a fork of ws4kp and runs in a web browser. As a refresher course in JavaScript I’ve made it my own by updating the interface to use new techniques such as Classes, and JavaScript standard replacements for jQuery. The project does not strive for accuracy as much as the Simulator does. This is much more of a no-fuss, just get a forecast project. I also made one addition that wasn’t present in the Weather Channel’s original hardware: an hourly forecast for the next 24 hours replaces the travel forecast in the default lineup. You can follow it and view the source on Github.

New hourly forecast display

The project’s name comes from the line of hardware that The Weather Channel provided to your local cable office to show the forecast. WeatherSTAR includes the acronym Satellite Transponder Addressable Receiver. As the acronym implies all of the data came from the satellite feed, and was hidden in clever ways right along the video broadcast. The first three units I, II and III were used throughout the 80’s and could only display text information, no icons with sun and clouds and no current radar. In 1990 the 4000 was introduced and was used for most of the decade. It featured graphics depicting current and foretasted weather and initially a still local radar image. A few years later it was updated to show several animated frames of the local radar. Wikipedia has a lot more information about the WeatherSTAR lineup.

I’m excited to announce I am releasing a long-running side project of mine https://temperature.express. It’s built to get you a forecast quickly in a graphic format.

Example Temperature Express forecast

The basic code for it is open source and available on Github. A few excerpts from its FAQ help to explain the project:

What is this?

The National Weather Service provides a huge amount of forecast and climate data for free including an hour-by-hour forecast for the next 7 days. Unfortunately, getting it displayed in a concise format is not something they do well. The main graph attempts to display the most relevant parts of the forecast (high and low temperature, wind chill, heat index, clouds, precipitation and dewpoint) in an easy to digest format.

Why is this better than my weather app?

This web site is meant to be fast, fast, fast. If you add a bookmark to this web site to your home screen I’d be willing to guess that this web page loads faster than your weather app and gets you the information you’re looking for 2-3x faster every time.

I chose an intentional minimal design to leave as much space as possible for the data. I’m an engineer, I prefer accurate information over cute, bubbly buttons.

Why a graph?

It makes it quicker to answer the typical weather questions like: Will it be warmer or cooler tomorrow? – Just compare the heights of the red line. Is it going to rain tomorrow? – Look for the blue filled areas. Will the sun be out on Saturday? – Look for no grey filled area on Saturday.

The graph makes it much easier to convey how the weather will change during the day. Your favorite weather app, or search result card might simply say “Sunny, High of 80” today. But that tiny little bit of information might mask something important like the temperature dropping to 60 by 5 pm. That big temperature drop might mess up your evening plans if you were going to be outside.

If you’re interested in the exact values, just tap any point on the graph to get the exact value and the time for that value. Or, If you want to view most of the data as a table, you can get one by clicking the table icon on the menu. I think you’ll quickly see that the graph is a much easier to digest format.

Why do you work on this?

I’m a programmer. JavaScript, HTML, Node.js and other web technologies are not part of my day job, but seem to be where the innovation is happening. I use this and other side projects to keep myself up to date with them.

Syntax is structure of the programming language. It can be thought of as the verbs, nouns and parts of speech that make up English, or a foreign language. Syntax can typically fall under 3 different categories: High level, low level, and graphical.

High level is what is most common today. This includes languages such as C, Visual Basic, PHP and others. These languages use human-readable elements such as if, then, else, while and return to control the flow of a program. Many of these languages share enough common elements that a programmer familiar with one language can often look at code written in a different language an often have an understanding of what it accomplishes. However, each of these languages is unique and many of them are designed around a particular task or programming concept. A thorough understanding of this underlying concept can help a programmer to make a decision on which language to use to accomplish a particular task.

A graphical language is a high level language that is often represented in a flowchart form to facilitate human readability. Flowcharts are mainly made up of decision branches, and function blocks, not unlike ones that are used to describe a process in a user manual. These languages are typically programmed using drag and drop. Grab the block you need and drop it in the appropriate place on the screen. Each block often has a few parameters associated with it. For example a decision block will have a two input values and a comparison operator, not unlike an if statement in a text based language. From this block you can then branch to a yes or no function block. Function blocks, like decision blocks, also have inputs. A block that performs and addition function, for example, would have two inputs for the values being added, and one output for the result. These languages often become very complex to follow because it is often difficult to place comments within the flowchart structure to help another programmer understand exactly what is happening, unlike comments in a text based program which can be placed nearly anywhere. Additionally, as the program becomes more complex, the flow of the program can become very difficult to follow as it will quickly begin to cover several pages, and connecting blocks between these pages can often be a challenge depending on exact language used.

Another example of a graphical language is ladder logic. It is typically used in industrial control systems, and is based directly on electrical schematics that look a lot like rungs of a ladder. This design was intentional, as an electrician familiar with common electrical symbols could very easily transition to programming in a ladder language, as the symbols and functions of the symbols was essentially the same. Over time, the language has evolved to add instructions and symbols that do not have an electrical counterpart. Ladder based languages also have the distinction of running continuously, unlike the text languages also discussed in this section. Again, this design is intentional as it is modeled after electrical schematics and components, which behave like a continuously running program, as long as the machine is turned on.

Low level languages are ultimately what is run on a processor, and the other two types of languages discussed here are ultimately turned into a low level language before they are executed. Low level languages often use mnemonics that help to make the language readable to programmers. A few lines from a typical low level language might read as follows:

LD A, 10
INC A
JP A,Z

In the above example LD, INC and JP are all opcodes, and are actual instructions that the processor carries out. The values following the opcodes are parameters for each code. In this case A is a register, 10 is a constant and Z is a label to jump to. Each of these opcodes is turned into a binary value that is the actual sequence of bits that the processor must see to carry out the command entered. Even though this has been made easier to read through the use of mnemonics, at first glance it is not entirely clear that this code loads the value 10 into register A, increments the value in A by 1 and then jumps to label Z if A is not zero. Programming in an assembly language requires a good understanding of the actual architecture of the processor, how memory is laid out, what functions it can perform and how to combine the rather primitive functions provided into more common functions such as if-then-else statements.