Release

Modify arscl1cloth/include/vip_config_extra.h to point to my source src/utilities/ dir

 

Copy over most recent QC for each site into etc/arscl_data.quality/ files; Check in these files.

 

Check in vip_config_extra.h.

 

Create arscl1cloth/etc/etc_setenv/etc_setenv.bnl_linux, etc., the BNL environment files

 

Check new env. files into RCS

 

co etc/etc_setenv/Makefile.cpp and add etc_setenv.bnl and other BNL environment files to AUX variable.

 

ci Makefile.cpp

 

Mark release with version number.

I believe there is a make command to do this, but for some reason I believe Eugene said it's not working right. Anyway, to have release number appear in output files, go to platform//cnvt/ dirs for each platform (only mpl and mmcr have output files) and enter:

find . -name "*.main.c" -print -exec grep "Version" {} \;

This shows all the routines that slap version id's into output files. Execute this rcs command on each of these files:

rcs -sRelease_4_0 -nRelease_4_0: 

replacing "Release_4_0" with appopriate release number.

exec the tcsh shell, if necessary (Linux):  exec /bin/tcsh

 

define VIP_CONFIG as    /home/kjohnson/arsclsource/arscl1cloth/include/ (or source arscl1cloth/etc/etc_setenv/etc_setenv.bnl..., if on approp. machine)

 

Build the etc/etc_setenv Makefile: make mf

NOTE: Will get a warning:   makedepend: warning:  cannot open "*.c"

   Ignore this.

Potential goof: When tried to do "make mf",  it died because VIP_CONFIG wasn't defined.  (and removed Makefile also).  Copied Makefile~ into Makefile , defined VIP_CONFIG as      /export/home/kjohnson/projects/arscl/arscl1cloth/include/ (since I was on cirrus) and did "make mf" successfully.

 

Ensure that the directories VIP_HOME and DATA_HOME exist

 

do the entire release build:

cd /home/kjohnson/projects/arscl/arscl1cloth/

make rcsclean

make checkout

source etc/etc_setenv/etc_setenv.bnl_linux

make mf

make (see Make Problems for details...)

make release (to get stuff into VIP_BIN)

Size of release on platforms varies:

Platform	Size of VIP_BIN
Solaris 2.6	97.8 Mb
Solaris 2.7, Sun disks	94.2 Mb
Solaris 2.7, PC disks	106 Mb
Linux	42.5 Mb

Data

Note that we cannot process data across boundaries in this file:

etc/arscl_instrument.config/arscl1.instrument.configuration

See this easy-to-read version of the arscl instrument configuration (as of 4/01).

Running

On cumulus, to run arscl1:

exec /bin/tcsh

Source etc_setenv.bnl_linux, if not done already.

Copy clear sky profile (<site>mpl*) files from VIP_BIN to VIP_HOME/tmp;  Change their permission to 666.

SGP 10/98 mwr format change: mwr problem.  In file /apps/vip/info/arscl_varlist.mwrloslilj.clearcloud_sgpmwrlosC1.a1.001.001 should have, prior to 10/98, 10(*).  After 10/98, need 20(*).

TURN OFF SCREEN SAVER

Be sure display is where we want it.  It can be set to any machine with an X server running.  This means you must be logged in on the display host machine.

Create data quality remarks file (e.g. VIP_HOME/data_quality_remarks.<site>.yyyy.mm)

Enter:

arscl1 -s 20001205.000000 -e 20001206.240000 <site> <Cx>
<path/to/data_quality_remarks.<site>.yyyy.mm> 0 >&! arscl1.output_<site>.yyyy.mm

where arscl1.output_<site>.yyyy.mm saves runtime output (e.g. in VIP_HOME/)

Are there data quality issues?

New Data Run:  Use ARSCL Data Quality Manual to assist in creating any needed DQ as input to ARSCL2.

Historical Run: Look at data quality db for this data (etc/arscl_data.quality/ the sgp file). There IS a data quality remark for 12/5.  Paste this into <data quality remarks file> specified in arscl1 run.

BE SURE THE DATA QUALITY FILE ENDS WITH A CARRIAGE RETURN

If it does not, arscl2 will consume entire CPU as it continues reading until it sees a CR. (It can be empty, but if there are one or more lines, must end in \n)

Run arscl2:

exec /bin/tcsh

Source arscl_setenv.bnl_linux, if not done already.

Set display.  It can be set to any machine with an X server running.  This means you must be logged in on the display host machine.

Run arscl2:

 arscl2 -s 20001205.000000 -e 20001206.240000 <site> <Cx> </path/to/data_quality_remarks.<site>.yyyy.mm> >&! arscl2.output_<site>.yyyy.mm

Run arscl3:

When satisfied with the results of arscl2's QC, run arscl3 to rename output files and move them to the arscldata area:

Create (if needed) the directories in $DATA_HOME/<site>/: (otherwise you will have to move data from the tmp area in by hand, plus the tar will --I think-- be incomplete). Note that the mmcrmode... directories will be created as needed by arscl3.

<site>arscl1cloth<Cx>.c1
<site>arsclbnd1cloth<Cx>.c1
<site>arsclcbh1cloth<Cx>.c1
<site>mplcmask1cloth<Cx>.c1
<site>mplsmask1cloth<Cx>.c1

Check that the arscl_csh.clouds.processing.parameters file in <VIP_HOME>/tmp/arscl_<site><facility>/ directory spans entire month's days (it could have been modified in rerunning parts of the month through ARSCL2). 

Run arscl3:

exec /bin/tcsh

Source arscl_setenv.bnl_linux, if not done already.

Set display.  It can be set to any machine with an X server running.  This means you must be logged in on the display host machine.

Run arscl3:

arscl3 -s 20001205.000000 -e 20001206.240000 <site> <Cx> </path/to/data_quality_remarks.<site>.yyyy.mm> >&! arscl3.output_<site>.yyyy.mm

Re-Running ARSCL

To rerun the whole thing cleanly, from the beginning:

Delete VIP_HOME/tmp contents (can leave arsclsgp)

Delete or move contents of output data directories (but keep all directories, and input data):

   $DATA_HOME/sgp/<output-dirs>

Remove contents of $FTP_HOME/pub/mmcr-vap/ directory

Repeat Running Section above.

To rerun just arscl2

(adding new DQ comment perhaps
see experience with sgp 2001.04 in mid June)

If we are limiting dates on rerun, need to

modify tmp/arscl_sgp/arscl_csh.clouds.processing.parameters

so it won't recompute entire month for some things. Specifically, you must modify  set period = "{200105}" and beg..., end... dates appropriately.  Note that 'period' is the match string used in selecting which files will be operated on by mpl (at least).  So for just one day, period could be set period = "{20010525}" and for two days, period would be
set period = "{20010525,20010526}"

Or, perhaps you could copy good days' data out before rerunning arscl2 (sounds like more work and I haven't tried this)

Note that subsequent arscl2 runs on same data seem to not produce all gifs in tar file in ftp area.  Specificallly, we seem to be missing files like:

    mplcamp/:  backscatter.gif, clearcloud.gif, cloudmask.gif
  mplcloth/:  background.gif, backscatter.gif
  mwrloslilj/:  clearcloud.gif, vapliqwetwindowxy.gif
  vceil25kstd/:  backscatter.gif, clearcloud.gif
These seem to be mostly input data gifs.  However, all the data seems to be there in quicklook area and arscldata/sgp area

Examining Output

FTP_HOME: Contains tar of images from runs.  Also has data quality file used.

DATA_HOME: has quicklooks/ directory and <site>/output data files

quicklooks/sgp/

Has some of same gifs as tar file but in directories with names of corresponding data  file

Corresponds to quicklooks available from www.ec.arm.gov/data/saql/

 

sgp/

Has all input and output arscl data

VIP_HOME/tmp: Contains lots of stuff (arscl_sgpC1/ and sgpmplC1.a1.20001217.000010.clr.final.cdf file)

Running Multiple ARSCLs Simultaneously

Running multiple ARM sites at once creates no problems--as separate tmp/arscl_<site><facility> directories exist.

Running multiple months at a single site simultaneously or starting a new ARSCL1 run before running ARSCL2 on previous month's data (at same site) does require planning.

To run more than one ARSCL at the same time (i.e. simultaneous processes running)

Specify separate VIP_HOME environment variables for each run, so VIP_HOME/tmp areas are separate.  VIP_BIN remains the same for both.

To interleave a new ARSCL1 run, while waiting on QC for previous ARSCL1 run:

After ARSCL1 finishes, rename tmp/arscl_<site><facility> to a temp name, until we are ready to run ARSCL2.  Then we can run a new month of ARSCL1.

Ship Data To Archive

We will use the XDC's site transfer capabilities.  I sent an e-mail to archive-notice@arm.gov letting them know about the data transfers.

Do the following:

Even though arscl1cloth files are large (84MB each), don't compress them.  The archive prefers to received uncompressed data.

Make the directory on arscl1.das.bnl.gov: /export/xferArchive/xfer<site><yyyymm>/

scp files to this directory.

Make sure permissions are 664 as this is what is required for the archive transfer directory on dev.xdc.arm.gov.

On dev.xdc.arm.gov,  check space on /prod/transfer/spool/c1.archive.arm.gov; tell Lynn if space is getting low.

 scp files to the /prod/transfer... dir on dev.xdc (again, ensure permissions are 664)

 The files will be picked up from here automatically and shipped to archive.

Ship the arscl1cloth files to the DQ office, as follows.

Ship arscl1clothC1.c1 files to Data Quality Office

Karen Sonntag at the DQ Office, requests that arscl1clothC1.c1 files (for ALL sites, beginning with data shipped after 5/27/2004) be sent to the DQ office as they are generated.  The files are sent to the DQO via the DMF with Karen Creel's help.

Be sure files have permissions 664.  Files can be gzipped or not.

  scp files to dev.xdc.arm.gov:/prod/transfer/spool/c1.dmf.arm.gov

Notify Karen Creel when files have been placed here.

Ship Quicklooks to DMF

Karen Creel needs to get all the quicklooks generated as final ARSCL output.  This is not being done currently.  We hope to move all ARSCL quicklooks to BNL in the future.

Tar up Images for Local Use

cd to quicklook area $DATA_HOME/quicklooks/<site>

tar -cvf     quicklooks.<site><Cx>.yyyymm.tar     */*yyyymm*

Move this tar to this site's arsclStore area: /<rootdir>/arsclStore/quicklooks/<site>/
(I seem to be storing these on cirrus in /data/arsclStore/quicklooks/<site>/ in addition to on local machine)

Put this on two dlt tapes, in its own file (see Putting Data on Tape section)

Put Data on Tape

• Mount arscl1.das.bnl.gov:/export/xferArchive/ directory on stratus at /export/xferArchvive (if not already mounted).
• Gzip <site>arscl* and <site>mpl* files.  Don't zip the <site>mmcrmode* files; they don't compress much and are very slow to do that.
• Follow directions in ~/dltTape/tapeAddProcedure to add these data to a dlt tape

Cleaning Up

Add new QC for this run to Data Quality database file in arscl1cloth/etc/arscl_data.quality/arscl_data.quality.remarks.database.<site><facility>

Remove contents of VIP_HOME/tmp when arscl2 output is accepted.
 

2.7Make Problems

Linux build: cc is link to gcc.  Makefiles use cc ... -R ... but cumulus complains that -R option doesn't exist (it doesn't, for gcc)

 
This is not a problem, because the only thing the -R construct adds is the path to the BW libraries, which ARSCL doesn't use anyway.  Numerous warnings are generated, but the build proceeds.

Bison problem:

Building in src/utilities/netcdf/cnvt/ requires 'bison.'  This is gnu freeware, which was part of the old arm environment.  Now, ARM officially only uses the Sun compiler, not gnu.   So they do not want  us to use bison, but rather yacc (which comes with the Sun environment).

I am modifying the Makefile.cpp to try to use yacc.

Change

bison -v -d convert.y

to

yacc -v -d convert.y

Change

$(CC) $(OPTIONS) -o cnvt convert.tab.c lex.yy.c -ly -ll

to

$(CC) $(OPTIONS) -o cnvt y.tab.c lex.yy.c -ly -ll

Try a make.

New problems:

File created by lex (lex.yy.c) tries to include 'convert.tab.h' which is no longer being created by bison.  It should be 'y.tab.h' instead now.  I must modify convert.l to include   y.tab.h.

It compiles, but with several warning errors (see todo file).  Doesn't compile if I try to use cc rather than gcc.

I see that this Makefile only uses gcc for CC.  All others use Sun C compiler, cc, as specified in include/vip_config.h.  I am getting other compile warnings also...

QC Quirks

• mpl BADDATA:  If a QC file has any mpl BADDATA qc in it, must run entire arscl2 run (over entire period covered by the run--typically one month) each time.  Cannot just rerun selected days.
• mmcr CLUTTERCORRECT:  Will not do clutter correct at more than one height range at a give time.
• mmcr Terrible job at cloud masking for one or (typically) more modes: see SGP 2003.12.22, 2004.06.09
• Look to see if debug prints from arscl_mmcrmom.msk.sgp.001.001.main.c
       indicate only 1 (or very few) noise points  went into mean
• If so, identify which times had very low noise values by running
      mmcrTools/mmcrAnomylousNoise.pro
      for suspect modes--see if odd noise  values appear for a few times (very low)
• Mark as bad the time identified above
• Run arscl2.
  
•