There were two separate problems with the March 2000 IOP MMCR data at the archive as of 11/2001.

First, a bit of history.  The data at the archive currently are actually the product of a reprocessing of the ingest that occurred in July 2000.  The reprocessing was made necessary by an inadvertent one-hour offset in the clock of the OS2 that occurred during a visit on 17 December 1999.  This offset persisted until March 22, 2000 at 1800.  The fix applied in July of 2000 was to add one hour (3600 seconds) the timestamps recorded in the raw data files.  This was done in the mmcr_ingest code.

However, there are gaps in the data sets that are not understood, which occurred only for the reproecessed data.  These gaps were not present (I hear, from others) prior to the reprocessing. The data for 2000.03.08 - 2000.03.10 have large spans of missing times.  In an attempt to understand and solve the problem we brought a copy of the MMCR ingest procedure to BNL and got it running here.  In the process of attempting to reproduce existing ingested files from the beginning of March in order to test the ingest at BNL, it was discovered that there was a mismatch between the number of records in the sgpmmcrcalC1.a1 and sgpmmcrmomentsC1.a1 files for most days from 3/1 - 3/10/2000.  This is something ARSCL does not like (although successful ARSCL runs were made on these days?!).

A record of MMCR data collected during the March 2000 IOP, both the archive and corrected BNL versions can be found in this document: MMCR Data fromMarch 2000 IOP

Here is a summary of the problems that were found, and their solutions.

• On most of the first 10 days of March 2000, there were (often repeated) data gaps in the 'cal' files, each lasting for exactly 30 minutes, or multiple of 30 minutes.  This corresponds to the length of one d, h, or Cal raw data file. After discussions with Krista Gaustad,  she recalled that there had been an error in a shared library file that existed at the time of the July 2000 ingest reprocessing effort.  There error probably had to do with a problem in the "build_filelist" routine, part of the 'sds_util' shared library.  Krista says the problems appeared when only when large numbers of files were being processed in one run. No one realized at the time that the error could have affected the MMCR data reprocess effort.  Apparently it did.  When I reran the ingests with the current ingest code and library environment, and adding the one hour offset, the resulting cal and moments 'a1' files had perfect agreement in record numbers.  As of 12/18, it looks like a build_filelist error that could have caused the problem was fixed 1/8/01 by Brian Ermold.  The problem was that when max files was reached, the currently read filenames were sorted and then returned.  It was fixed to read in all file names in the directory, sort them all, and then chop at the maxfiles point.
• There were additional problems with data from 2000.03.08 - 10.  There were actually two versions of these raw data at the archive: v0 and v1.  The v1 data was presumably used in reprocessing the ingests and that is what we initially used at BNL also.  We discovered that there were additional, unrelated raw files tacked onto these raw tar files.  The original files were tarred on March 8,9, or 10. But then, on March 23, additional d, h, Cal, mon, and pow files were added to the tar, with times interleaved with those of the original files.  On inspecting the Cal files using ncdump, we found that the parameters used differed from those of the regular Cal files for this month.   We retrieved the v0 versions of these raw files, and found that these extraneous files weren't there.  When the ingests were rerun with v0 files, perfect record number agreement was found!

For directions for running the MMCR ingest at BNL, as done during these tests, see Running MMCR Ingest Code at BNL.

Current State of mmcrcalC1.a1, mmcrmomentsC1.a1 files:

2/6/03

On February 6, 2003 the new sgpmmcrcalC1.a1 and sgpmmcrmomentsC1.a1 files were shipped to the archive as new versions for the following days: 1, 2, 3, 4, 5 and 8, 9, 10.   These files were created (in November 2001) at BNL by reingesting the raw MMCR data as described above and then running the sgpmmcrcalC1.a1 data files through the mmcrcal VAP to apply the 3.2dB offset and the (old) nearfield correction.  Note that these files are written using the new MMCR DOD.  The only differences between this format and the one exiting prior to 4/1/01 is in the names of (primarily global) attributes.  No field names were changed.

3/10/03

In some cases the two files for a given day were replaced by a single, corrected file.  We
requested that the files that were not 'covered' by new version be deleted from the archive.
Here is a list of files to be deleted and the reasons for the deletion:
 

File to Remove	Reason for Removal
sgpmmcrcalC1.a1.20000309.000100.cdf	This file is corrupted with extraneous files from another day.  This file, as well as a second corrupted file for this day, were replaced by sgpmmcrcalC1.a1.20000309.000113.cdf on 2003.02.07.   The second corrupted file for this day has the same name as the replacement file so it should not be inadvertantly retrieved.
sgpmmcrcalC1.a1.20000310.000055.cdf	This file was corrupted with extraneous files from another day. This file, as well as a second corrupted file for this day, were replaced by sgpmmcrcalC1.a1.20000310.000123.cdf on 2003.02.07.   The second corrupted file for this day has the same name as the replacement file so it should not be inadvertantly retrieved by a user.
sgpmmcrmomentsC1.a1.20000301.010000.cdf	This file has data gaps and so was replaced by the reingested file sgpmmcrmomentsC1.a1.20000301.000038.cdf on 2003.02.07.  A second file with missing data was also replaced by this file, but it has the same name as the replacement file so it should not be inadvertantly retrieved by a user.
sgpmmcrmomentsC1.a1.20000309.000100.cdf	This file is corrupted with extraneous files from another day.  The file was replaced by sgpmmcrmomentsC1.a1.20000309.000113.cdf on 2003.02.07.
sgpmmcrmomentsC1.a1.20000310.000055.cdf	This file is corrupted with extraneous files from another day so it was replaced by sgpmmcrmomentsC1.a1.20000310.000123.cdf on 2003.02.07.   A second file with missing data was also replaced by this file, but it has the same name as the replacement file so it should not be inadvertantly retrieved by a user.