IPP Progress Report for the week YYYY.MM.DD - YYYY.MM.DD

(Up to IPP Progress Reports)

Eugene Magnier

Serge Chastel

Heather Flewelling

Roy Henderson

• Put PSPS loading data into czartool for easier monitoring.
  • removed revision/db upgrade stuff from czartool: not necessary, as I will maintain a dump of schema shape instead.
  • added tables to CzarDb of same shape as IPP ones: 'pending', 'processed', 'faults' for each stage
  • changes to ippToPsps metrics program to write new czarDb tables
  • adapted czarplot to deal with ipptopsps stages
  • put ipptopsps metrics program on a timer so it can routinely write to CzarDb
  • updated czardb maintenance code to trim and optmize new tables
  • updated czarpoll to produce ippToPsps plots
  • created new PHP page in czartool for display

• Duplicates broke PSPS merge. Investigated and found that duplicate frames were in DVO. Helped Sue remove them before re-merging. This prompted some extra duplicate protection measures:
  • created ipptopsp 'test' database to reduce confusion when I'm testing new code
  • increased 'processing now' limit from 2 to 4 hours, as when network is slow it is feasible that a batch could take longer than 2 hours
  • removed 'loaded_to_datastore' constraint when checking if a stage_id has already been processed as it is a little dangerous (could be called just after a batch is processed by one client, but before it uploads that batch to the datastore)

• Other ippToPsps coding:
  • found bug in loading code: batches that failed at the tar and zip stage prior to publication to datastore were being labelled as successfully processed, then failing to publish
  • populated some new fields for stacks

• Other
  • my crontab'd backup script for various Dbs (ipptopsps, czadb, mailing lists) now dumps a copy of the schema as well
  • cleaned-up some failed batches
  • work setting-up re-loading of SAS data to PSPS

Mark Huber

Bill Sweeney

Chris Waters