Ingestors

An ingestor is a program that reads data entering the system through a communications port. Ingestors read asynchronous data from conventional data circuits such as DDS, IDS and PPS.

Each circuit has a text formatted configuration file that the ingestor reads to configure the communications port. This configuration file resides in ~oper/mcidas/data and is usually named with the circuit name followed by .CFG, for example, DDS.CFG. It contains information such as baud rate and the number of data bits or stop bits.

McIDAS-XCD supports these types of ingestors:

Text ingestors

A text ingestor receives data from the NOAAPORT broadcast in ASCII format. Each incoming circuit has its own text ingestor that writes to a set of raw text files and index files. Index files contain the location information of data in the raw text file.

For example, Figure 1 shows that the text ingestor INGETEXT ingests data from NOAAPORT. Each ingested data block is placed in a circuit-specific raw text file for that day. INGETEXT also files information about the data block into an index file which is used by text applications and decoders for locating data quickly.

The naming convention for the circuit-specific raw text file is ccyyddd0.XCD, where cc is the first two characters of the circuit name, and yyddd is the Julian day. The naming convention of the index file is hhyyddd.IDX or hhhyyddd.IDX where hh is a 2-character WMO header, hhh is a 3-character AWIPS header and yyddd is the Julian day.

While only one text ingestor can write into a text file, any text ingestor can write to any index file. For example, terminal forecasts (FTs) arrive on the DDS and IDS circuits. The actual forecasts are filed in DDyyddd0.XCD and IDyyddd0.XCD, respectively. However, both ingestors write their directory information into the same index file, FTyyddd.IDX. This ensures that applications will work consistently on similar data formats regardless of the data's source.

Index files also store related data that arrives under more than one WMO header. For example, mandatory upper level RAOB reports come in under the headers UJ, US, UK, UL, etc. Rather than having a separate index file for each header, a routing table is created during installation to tell the ingestors where to file the headers from each circuit. For example, the index file for all RAOB WMO headers is UJyyddd.IDX. If a WMO header is not forced into a particular index file, it is filed in the ZZyyddd.IDX file, which is a miscellaneous index.

Binary ingestors

The binary data ingestor, INGEBIN, ingests a binary data stream regardless of the data format. INGEBIN writes the data to a circular spool file, HRS.SPL, where it can be processed by a data monitor. The data that INGEBIN ingests includes model output grids in GRIB1 or GRIB2 format, NEXRAD radar data in its native format, and various text products in BUFR format. For further information see Chapter 2, Configuring the NEXRAD Decoder, and Configuring the GRIB Decoders.

Figure 1. Conventional Data Ingesting Flow Diagram

[Search Manual] [Table of Contents] [Go to Previous] [Go to Next]