McIDAS-X User's Guide
Version 2022.1

[Search Manual] [Table of Contents] [Go to Previous] [Go to Next]

DSSERVE

Manages and lists dataset names on the local server.

Formats

DSSERVE ADD dataset format bfile efile [keywords] "comment
DSSERVE DEL dataset
DSSERVE LIST group format

Parameters

Keywords

General Remarks

The dataset parameter must be specified in the group/descriptor format. The total length of dataset, including the slash, must be 21 characters or less. The group name is limited to eight characters or less and the descriptor name is limited to 12 characters or less. The group name cannot be entirely numeric. The following characters are not valid in the group or descriptor name: slash ( / ), period ( . ), blank ( ), left bracket ( [ ), right bracket ( ] ).

The ADD option overwrites an existing ADDE dataset name without warning. This means you cannot assign the same ADDE dataset name to more than one dataset, even if the datasets have different file formats.

After adding the dataset name, the ADD option maps the dataset's group name to the local server in the DATALOC table. If the group name is already mapped to a remote server, the DATALOC table entry is not changed.

DIRFILE Keyword Remarks

Use the DIRFILE='mask' keyword to access non-McIDAS format files or McIDAS area, grid or MD files with names other than AREAnnnn, GRIDnnnn and MDXXnnnn (where nnnn is a 4-digit number) and/or files outside the MCPATH and REDIRECT directories. The mask value contains the location and names of the files (i.e., the directory and file masks), and is specified using standard Unix substitution rules and wildcards, except that the apostrophe ( ' ), comma ( , ), curly brackets ( {} ), and equal sign ( = ) characters are not valid. The file mask consists of all characters after the last slash; everything before and including the last slash is the directory mask. For example, if you specify DIRFILE='/home/user/goes/images/199?/Conus*', then /home/user/goes/images/199?/ is the directory mask, Conus* is the file mask, and the entry matches all files beginning with "Conus" that reside in any /home/user/goes/images/ subdirectory whose name is four characters long and begins with "199". If you don't specify any slashes in mask, there is no directory mask so the entry is treated as a file mask only and its file(s) must be located in a MCPATH directory.

You can verify that the mask you specify in DIRFILE='mask' is correct (i.e., that the file[s] are accessible from McIDAS-X) by running a Unix ls command with the mask in the ! (Exclamation Point) command from your McIDAS-X session. For example, if your DSSERVE command includes DIRFILE='/home/point/netcdf/data/RAP*.nc', run command !ls /home/point/netcdf/data/RAP*.nc to confirm that McIDAS-X has access to the RAP*.nc files. Similarly, if your DSSERVE command also includes the INFO= keyword pointing to a file, you can use the same technique to check that it is accessible (e.g., if DSSERVE includes INFO='/home/point/netcdf/config/RAP.cfg', run !ls /home/point/netcdf/config/RAP.cfg).

Absolute position numbers in datasets created using the DIRFILE keyword are determined in the order that the files are found by the server, beginning with position 1. Thus, the absolute position number of a file can change if another file is added to or removed from the dataset. Relative position numbers work as expected, so position 0 is the most recent image, -1 is the next most recent, etc.

AIRS HDF Dataset Remarks

You can access Aqua AIRS Level 1b HDF files with the IMG* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "AIRS" in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword. The file names must match the standard naming convention that includes the 5-minute granule number (JPL format).

COMS HRIT Dataset Remarks

You can access COMS HRIT files with the IMG* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "COMS" in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword. The file names must match the KMA segmented image file naming convention, IMG_ppp_08_cccc_YYYYMMDD_hhmmss_nn.hrit or IMG_ppp_08_cccc_YYYYMMDD_hhmmss_nn.hrit.xrit, where ppp is the projection (FD, ENH, LSH, APNH or LA), cccc is the spectral channel (VIS, SWIR, WV, IR1 or IR2), YYYYMMDD_hhmmss is the date and time, and nn is the segment file number (01-10).

GeoTIFF Dataset Remarks

GeoTIFF images are similar to TIFF images, but include calibration and navigation information. They cannot be read in McIDAS-X but can be in McIDAS-V, ARCVIEW and other GIS software packages.

You can create GeoTIFF image files from other TYPE=IMAGE datasets. To do so, make a GeoTIFF image dataset with a DSSERVE command similar to: DSSERVE ADD GEOTIFF/GOES GEOT 1 9999 TYPE=IMAGE "GeoTIFF GOES image files. Then specify GEOTIFF/GOES as the destination dataset with a positive position number in an IMGCOPY, IMGFILT, IMGOPER or IMGREMAP command. If you use IMGCOPY, IMGFILT or IMGOPER, the source image must be in a rectilinear projection. If you use IMGREMAP, the source image can be in any projection, but you must specify PRO=RECT to make the destination GeoTIFF image a rectilinear projection. IMGFILT, IMGOPER, and IMGREMAP save the image with BRIT calibration. IMGCOPY can save it with other calibrations by using the UNIT keyword.

By default, the data values in output GeoTIFF files are stored as scaled integers. To store them as 32 bit floating point values, use the INFO= keyword to point to a text file whose content is "FORMAT=FLOAT" (one line, without the quotes). For example, if you include keyword INFO='/home/username/mcidas/data/float.txt' in your DSSERVE command and float.txt contains just FORMAT=FLOAT, the GeoTIFF files written to the dataset will contain data stored as 32 bit floating point values.

The output file name consists of the descriptor name and four-digit position number followed by .tif. For example, the command IMGCOPY GOESEAST/VIS GEOTIFF/GOES.300 SIZE=ALL creates an output file named GOES0300.tif.

GOES-R Series netCDF Dataset Remarks

You can access GOES-R Series (GOES-16 and later) netCDF image files with the IMG* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "ABIN" in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword. The files must be mission-standard Level 1b files in netCDF-4 format (like those from NOAA CLASS or decoded from the GRB data stream using CSPP Geo) or mission-standard Level 2+ products in netCDF-4 format (like those from NOAA CLASS or processed files from NOAAPORT [with headers and footers stripped and the tiles stitched together by software like Unidata's ldm-alchemy package]). The names of files must look like:
ABI-L1b-RadC-M3C01_G16_s2015229195720.nc (former CSPP Geo naming convention)
or
OR_ABI-L1b-RadC-M4C16_G16_s20151702215532_e20151702220362_c20151702220394.nc
or
OR_ABI-L2-CODC-M6_G16_s20191961731348_e20191961734121_c20191961736053.nc
OR_ABI-L2-CMIPC-M6C02_G16_s20191961731340_e20191961731340_c20191961731340.nc
(GOES-R Ground System naming convention).

You can access GOES-R Series (GOES-16 and later) netCDF Geostationary Lightning Mapper (GLM) point files with the GLMDISP, GLMLIST and PT* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "GLMN" in the format parameter, TYPE=POINT, the directory and file masks to the netCDF point files in the DIRFILE keyword, and the configuration file that maps the netCDF file parameter names to McIDAS parameter names in the INFO keyword. The files must be mission-standard format and follow the GRB naming convention, and thus look similar to OR_GLM-L2-LCFA_G16_s20172151749200_e20172151749400_c20172151749467.nc. Three configuration files for GLM data are provided in the ~mcidas/data directory: GLM_EVENT.cfg, GLM_FLASH.cfg and GLM_GROUP.cfg.

GRIB Dataset Remarks

You can access GRIB files in GRIB1 or GRIB2 format with the GRD* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "GRIB" in the format parameter, TYPE=GRID, and the directory and file masks in the DIRFILE keyword. Only GRIB1 and GRIB2 files from the NOAAPORT data stream have been tested and are fully supported.

Himawari Dataset Remarks

You can access Himawari HSD (Himawari Standard Data) format files with the IMG* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "WARI" in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword. The file names must match the JMA HSD image file naming convention, HS_aaa_yyyymmdd_hhnn_Bbb_cccc_Rjj_Skkll.DAT, described in the Himawari Standard Data User's Guide.

You can access HimawariCast HRIT files in the JMA standard (segmented or combined) format with the IMG* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "WARC" in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword. The names of files should look similar to IMG_DK01B16_201508290810_010 or IMG_DK01IR2_201601120130.

JPSS Series Dataset Remarks

You can access Joint Polar Satellite System (JPSS) Series VIIRS SDR files in HDF5 format with the IMG* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "VIIR" in the format parameter, TYPE=IMAGE, the directory and file masks to the data files in the DIRFILE= keyword, and the directory and file masks to the geolocation files in the INFO= keyword. Also, the data (SVM*, SVI*, SVDNB*) and terrain-corrected geolocation (GMTCO*, GITCO*, GDNBO*) must be in separate files in the NOAA CLASS naming format. Examples:
SVM05_npp_d20170821_t1823015_e1828419_b30140_c20170822002843330215_nobc_ops.h5
GMTCO_npp_d20170821_t1823015_e1828419_b30140_c20170822002842363247_nobc_ops.h5
SVI01_j01_d20180310_t1930133_e1931378_b01595_c20180310201520720740_nobc_ops.h5
GITCO_j01_d20180310_t1930133_e1931378_b01595_c20180310201845788134_nobc_ops.h5
SVDNB_j01_d20181217_t0703033_e0704278_b05589_c20181218200444269328_nobc_ops.h5
GDNBO_j01_d20181217_t0703033_e0704278_b05589_c20181218200444269328_nobc_ops.h5

Kalpana HDF5 Dataset Remarks

You can access Kalpana Level 1b HDF5 files with the IMG* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "KPH5" in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword.

MODIS HDF Dataset Remarks

You can access MODIS Level 1b HDF files with the IMG* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "MODS" in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword.

You can access MODIS Level 2 MOD06 (cloud top properties), MOD07 (atmospheric profile) and MOD35 (cloud mask) products in HDF format with the IMG* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "MODX" in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword.

You can access MODIS Level 2 MOD04 (aerosol), MOD28 (sea surface temperature), and MODR (corrected reflectance) products in HDF format with the IMG* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "MOD4" (for MOD04 aerosol products), "MOD8" (for MOD28 sea surface temperature products), or "MODR" (for MODR corrected reflectance products) in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword.

You can access MODIS (and potentially other) HDF files as point data in MODS-format datasets created using DSSERVE with the TYPE, DIRFILE and INFO keywords. The TYPE keyword specifies the data type (POINT) of the HDF files in the dataset. The DIRFILE keyword specifies the directory and file masks to locate the HDF files. The INFO keyword specifies the name of the configuration file that maps the HDF file parameter names to McIDAS parameter names. The configuration file must be tailored specifically to the HDF files in the dataset. Two sample configuration files, named MOD06.cfg and MOD07.cfg, are available in the ~mcidas/data directory. Refer to those files when creating configuration files for your HDF point datasets.

When creating MODIS HDF image or point datasets, the file names must match the standard Goddard DAAC (Distributed Active Archive System) or MODIS direct broadcast naming conventions. In the DAAC convention, for example, 1 km Terra MODIS Level 1b HDF image files must follow the format MOD021KM.Ayyyyddd.hhmm.002.yyyyddd.hdf. Aqua files must begin with MYD instead of MOD, and 1/2 km and 1/4 km resolution files are denoted as HKM and QKM instead of 1KM. In the MODIS direct broadcast convention, for example, the same 1 km Terra image files must follow the format t1.yyddd.hhmm.1000m.hdf. Aqua files must begin with a1 instead of t1, and 1/2 km and 1/4 km files are denoted as 500m and 250m instead of 1000m. Similarly, these two naming conventions can be applied to the MODIS Level 2 products in HDF format.

MSG Dataset Remarks

You can access real-time native format Meteosat Second Generation Level 1.5 files transmitted by EUMETSAT with the IMG* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "MSGS" in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword.

You can access real-time segmented Meteosat Second Generation Level 1.5 files transmitted by EUMETSAT with the IMG* commands by using DSSERVE to assign a dataset name to the files. Compressed and uncompressed Level 1.5 files for the prime MSG service and rapid scan MSG service are supported. In the DSSERVE command, specify "MSGT" in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword. Also, the dataset descriptor must be "HRV" if the dataset contains high resolution visible channel (band 12) data, and either "FD" or "VISIR" if the dataset contains full disk low resolution visible and infrared channel (bands 1-11) data. For each image time, the data file(s) and corresponding prologue file (which contains information about the starting scan) must be present in the same directory; the corresponding epilogue file (which contains information about the ending scan) is not required but can also be present. The DIRFILE keyword's file mask must point to either the prologue file (for the server to return any image, whether or not it's complete) or the epilogue file (for it to return only images that are complete). For example, if you have the data, prologue and epilogue files in the /data/msg directory the following commands make a high resolution visible channel dataset using the epilogue files and a full disk low resolution visible and infrared channel dataset using the prologue files:
DSSERVE ADD MSG/HRV MSGT TYPE=IMAGE DIRFILE=/data/msg/H-000-MSG1*EPI* "MSG 1km broadband visible data
DSSERVE ADD MSG/FD MSGT TYPE=IMAGE DIRFILE=/data/msg/H-000-MSG1*PRO* "MSG 3km full disk imager data

MTSAT HRIT Dataset Remarks

You can access MTSAT HRIT files with the IMG* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "MTST" in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword. The file names must match the JMA segmented image file naming convention described at https://www.ssec.wisc.edu/mcidas/doc/sdi_man/current/mtsat_hrit-3.html.

netCDF Dataset Remarks

You can access netCDF grid and point files in NCDF-format datasets created using DSSERVE with the TYPE, DIRFILE and INFO keywords. The TYPE keyword specifies the data type (GRID or POINT) of the netCDF files in the dataset. The DIRFILE keyword specifies the directory and file masks to locate the netCDF files. The INFO keyword specifies the name of the configuration file that maps the netCDF file parameter names to McIDAS parameter names. The configuration file must be tailored specifically to the netCDF files in the dataset. Two sample configuration files, named netcdfgrid.cfg and netcdfpoint.cfg, are available in the ~mcidas/data directory. Refer to those files when creating configuration files for your netCDF grid and point datasets.

You can create netCDF-3 image files from other TYPE=IMAGE datasets. To do so, make a netCDF image dataset with a DSSERVE command similar to: DSSERVE ADD NETCDF/GOES NCDF 1 9999 TYPE=IMAGE "netCDF GOES files. Then specify NETCDF/GOES as the destination dataset with a positive position number in an IMGCOPY, IMGFILT, IMGOPER or IMGREMAP command. The output file name consists of the descriptor name and four-digit position number followed by .nc. For example, the command IMGCOPY GOESEAST/VIS NETCDF/GOES.300 SIZE=ALL creates an output file named GOES0300.nc that's mapped to position 300 of the dataset NETCDF/GOES. Note that these netCDF-3 image files cannot be read in McIDAS-X but can be in McIDAS-V.

POES and Metop Level 1b Dataset Remarks

You can access Metop FRAC AVHRR Level 1b or POES/NOAA AVHRR Level 1b files downloaded from NOAA CLASS (http://www.class.noaa.gov) with the IMG* commands by using DSSERVE to assign a dataset name to the files. When doing so, specify "LV1B" in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword. To process the data (during client command data requests) using the file's line-by-line lat/lon navigation (rather than the orbital navigation at the beginning of the file), also include the keyword INFO=LALO. You can compare the two navigation types by using DSSERVE to make two datasets that are identical except that one includes INFO=LALO.

The two types of AVHRR Level 1b files that can be used are 10-bit files (which inherently have all channels) and 16-bit files that contain all channels. You can obtain 16-bit files with only one channel from CLASS, but they won't work in McIDAS ADDE servers. In either case (whether you choose 10-bit or 16-bit), the file must contain the entire pass, not a subset of it.

You can copy POES/NOAA AVHRR data in McIDAS readable format (McIDAS Area with 192 byte doc section) to the CLASS Level 1b file format with IMGCOPY by using DSSERVE to assign a dataset name to the files. When doing so, specify "LV1B" in the format parameter, TYPE=IMAGE, and the directory and file masks in the DIRFILE keyword. The output file name consists of the descriptor name and four-digit position number followed by the extension .l1b. For example, the command IMGCOPY WALLOPS/N17HRPT LV1B/17HRPT.2 SIZE=SAME DOC=YES BAND=ALL creates an output file named 17HRPT0002.l1b in the directory specified in the DIRFILE keyword. The IMGCOPY command must include SIZE=ALL or DOC=YES with BAND=ALL and SIZE=SAME or SIZE= with any value for its lines parameter and either 2048 (for HRPT or LAC data) or 409 (for GAC data) for its ele parameter.

System Navigation Dataset Remarks

When adding a SYSN-format dataset, the descriptor portion of the dataset name must exactly match a satellite name in the text file ~mcidas/data/CORE.SAT. The server uses this file to map the descriptor to the correct sensor source number and locate the needed information in the dataset's system navigation files. For example, NAV/GOES9, NAV/GOES-9, NAV/G9, NAV/G-9 and NAV/WEST are all valid dataset names because all the descriptors are entries in the CORE.SAT file. You cannot add new satellite names or edit existing names in CORE.SAT; you must use a satellite name that exists in the SSEC-supplied version of the file.

Text Dataset Remarks

When adding a TEXT-format dataset, you must specify the text file name with the INFO keyword. The file name is case-sensitive. If you do not include the full path name, the file must be located in a MCPATH or REDIRECT directory. For example, you can create a TEXT-format dataset containing a NORAD Two-Line Element (TLE) file, then list and display its data with the NAVCALC and NAVDISP commands. To do so, place the TLE file into a MCPATH or REDIRECT directory, then make the dataset with a command similar to: DSSERVE ADD NAV/MODIS TEXT INFO='TLE.TXT' "TLE file with TERRA and AQUA orbits.

Examples

DSSERVE

This entry lists information about all datasets on the local server.

DSSERVE ADD LOCAL/AREAS AREA 1 9999 "Areas in my data directory

This entry assigns the dataset name LOCAL/AREAS to areas 1 through 9999 on the local server.

DSSERVE ADD NAV/NOAA-14 SYSN 1 6 "NOAA-14 System Navigation

This entry assigns the dataset name NAV/NOAA-14 to system navigation files 1 through 6 on the local server.

DSSERVE ADD BOB/GOES-WV-8KM AREA 6001 6100 "Hurricane Bob images

This entry assigns the dataset name BOB/GOES-WV-8KM to areas 6001 through 6100 on the local server.

DSSERVE ADD ADMIN/SATDATA TEXT INFO='/u/oper/mcidas/info/satdata.txt' "Listing of all available satellite data

This entry assigns the dataset name ADMIN/SATDATA to the text file /u/oper/mcidas/info/satdata.txt. Use the READ command to list the file.

DSSERVE ADD GOES/VIS_4KM HDF TYPE=IMAGE INFO='/users/ingest/hdfdata' "GOES-8 HDF data

This entry assigns the dataset name GOES/VIS_4KM to HDF image data on the local server. The directory specified with the INFO keyword is used by the locally-developed HDF server to locate the data.

DSSERVE LIST LOCAL

This entry lists information about all datasets in the group LOCAL on the local server.

DSSERVE DEL BOB/GOES-WV-8K

This entry deletes the dataset name BOB/GOES-WV-8K from the local server, but does not affect the data to which the dataset name was assigned.

DSSERVE ADD GRID/NETCDF NCDF TYPE=GRID INFO='NCDF_ETA.CFG' DIRFILE='/home/user/netcdf/grids/ETA*'

This entry assigns the dataset name GRID/NETCDF to netCDF grid files whose names begin with ETA and are located in the /home/user/netcdf/grids/ directory. The INFO keyword specifies the configuration file (NCDF_ETA.CFG) that maps the netCDF file parameter names to McIDAS parameter names.

[Search Manual] [Table of Contents] [Go to Previous] [Go to Next]