Thoughts on ARSCL Production Planning

7/23/01

Data Storage Requirements

ARSCL Run Products
 
DATA_HOME:
519 Mb / 2 days --> 7.8 Gb / month
VIP_HOME/tmp:
  819 Mb / 2 days --> 12.3 Gb / month        (Robin says 5 - 15Gb / month)
Total:
20 - 23 Gb per month
(stratus has 32Gb drive--could store 1 month there)

Time Required

SGP observed timings:
2001.04 - arscl1 12 hrs;    arscl2 12 hrs
2001.05 - arscl1 21 hrs;    arscl2 20 hrs
        (pc disks)
Manus Backlog:
stratus requires 2 - 3 months to do Manus backlog  (dual 360 MHz cpus)
(a bit longer if we start operations processing during that time)

Processing 2 days of sgp data on stratus:

1 hour in arscl1;
45 min in arscl2 (with only one data quality issue).

Scaling that up, 1 month would take: 15 hours in arscl1, and ~15 hours in arscl2 --> 30 hours of stratus time.  Plus evaluation of output and transfer to archive of xxGb data.  So, allow 3 days per month of data.  So 1.6 months get done per week.

For approx 24 months of data,  stratus would take up to 16 weeks = 4 months to process Manus backlog.

Operational Running:
Allow 3 days per month, per site for operational runs, including data acquisition  (plus arsclstat, merged sounding).

Hardware Requirements

Need multiple machines:
  • To get through backlog quickly
  • To do timely monthly processing from 4 sites (machines only needed part of the month)


    • Cirrus/Stratus Specs

    Sun Ultra60s
    Dual 360MHz CPU
    512 Mbytes RAM
    9 and 36 GB Internal Hard Drives
     

    Software/ Libraries Required

    Solaris 2.7
    gnu compiler (for making Makefiles)
    gunzip (if data zipped)
    netcdf
    X11 library
    (don't need idl currently)

    FTP

    Need an ftp directory where we can put quicklooks and intermediate data files for Eugene; Is it sufficient to give him an account on stratus?
     

    Input Data Acquisition

    One month of data comprises ~2.5-3.5 Gb / site.

    Manus data: 24 months X 2.5 Gb  ==> 60Gb

    Archive prefers ftp, as less expensive in time and resources.
    Tape available: 4mm, 8mm, DLT, and IBM 3590 formats

    Transfer Results to Archive

    Need to transfer ~5Gb data (1.5 Gb compressed) to archive per month of data processed, via ftp I assume.
    Operationally, 1.5Gb / month / site.
     

    Processing Status

    TWP.C2 (Nauru)

    Complete  11/98 thru 9/00
    Awaiting reprocessed MPL 10/00 -12/00

     

    NSA.C1

    Complete 11/98 thru 9/00, 
    12/00 - 3/01
    Awaiting reprocessed MPL data 10/00- 11/00
    Thru arscl1, awaiting Eugene's QC   

    SGP.C1

    Complete 11/96 - 4/01
    Thru arscl2,  awaiting Eugene's Review 5/01

    TWP Input Data Availability

     
    Input Data Stream
    Manus (C1)
    Nauru (C2)
    mmcrcalCx.a1 6-12/99
    2000    		 10-12/98
    1999
    2000
    mmcrmomentsCx.a1 6-12/99
    2000    		 1999
    2000
    mplnor1campCx.c1 no 98, no 99, no 2000 11-12/98
    1-11/99
    no 2000
    mplCx.a1 2-12/97
    1998
    1-6/99
    no 2000    		 11-12/98
    1999
    part 1/00
    smet60sCx.a1 1997
    1998
    1999
    2000    		 10-12/98
    1999
    2000
    vceil25kCx.a1 1998
    1999
    2000    		 11-12/98
    1999
    2000
    mplcbh1scottCx.c1 no no