Introduction

This is just a quick summary of MD PV2 into PSPS

• all MDs have been created in dvo, calibrated, and loaded onto the datastore.
• as of 9-26-2014, 2,3,4,5,6 are ready to be queried
• as of Oct 2014, all are ready to be queried.

Summary

This is a quick summary of what is below. I have investigated using 3 different methods the data integrity of the MD fields. Details are given below, but here are the results:

• Method 1 - plots of coverage of random ra / dec
• Method 2 - plots of coverage of random ra / dec / batchid
• Method 3 - comparison of counts in the ipptopsps database to counts in the psps database

MD field	Detection Table	Stack Detection	Object	Comments
	used methods 1 & 3	used methods 1 & 3	used methods 1, 2 & 3
MD01	missing detections	complete	missing
MD02	complete	complete	looks complete, need to verify using method 3
MD03	complete	complete	looks complete, need to verify using method 3
MD04	complete	complete	looks complete, need to verify using method 3
MD05	missing detections	complete	missing
MD06	complete	complete	missing
MD07	duplicates present	duplicates present	missing
MD08	complete	complete	missing
MD09	complete	complete	missing
MD10	complete	complete	missing

bold is used for those parts that are complete (no problems detected)

Plots of coverage

I did a selection of random 100,000 ras and decs from detectionFull, stackDetectionFull, and Object for each field. The results are plotted below, you can click on the image to make it bigger.

the queries I used for each MD (on production cluster):

select top 100000 ra, dec from detectionFull order by newid() 
select top 100000 ra, dec from stackDetectionFull order by newid() 
select top 100000 ra, dec from Object order by newid() 

The results (see images below):

MD01 DetectionFull, MD02 DetectionFull have stripe-like problems MD07 StackDetectionFull has a stripe-like problems

md field	detectionFull	stackDetectionFull	object
md01
md02
md03
md04
md05
md06
md07
md08
md09
md10

Plots with batchid or mag

There is a problem where a number of object batches failed to be made. The black/blue colors show the missing holes for the Object Tables

select top 100000 ra, dec, objid from object order by newid()

MD field	plot of batch id	comments
MD01
MD02		no holes
MD03		no holes
MD04		no holes
MD05
MD06
MD07
MD08
MD09
MD10

Comparisions of ndetections

Another way to check the integrity is to do comparisons of the number of detections for each batch. in the ipptopsps.batch table, there is a column (ndetections) that is the number of detections published (per batch). The following queries make counts of the number of detections for each of stack/detection and grouped by batchid:

select batchid, count(objid) INTO mydb.[md06detcounts2] from Detection join ImageMeta on Detection.imageID = ImageMeta.imageID join FrameMeta on FrameMeta.frameID = imageMeta.frameID group by batchid

select batchid, count(*) INTO mydb.[md01stackdetcounts] from StackDetectionFull join StackMeta on StackDetectionFull.StackMetaID = StackMeta.StackMetaID group by batchid

I wrote a perl script to compare the numbers between these 2 databases. I find the following

MD field	Detections	Stacks	Objects
MD01	1565 out of 3797 batches are missing detections	all good	not done yet
MD02	all good	all good	not done yet
MD03	all good	all good	not done yet
MD04	all good	all good	not done yet
MD05	2236 out of 3836 batches are missing detections	all good	not done yet
MD06	all good	all good	not done yet
MD07	4066 out of 4066 batches have extra detections	164 out of 321 batches have extra detections	not done yet
MD08	all good	all good	not done yet
MD09	all good	all good	not done yet
MD10	all good	all good	not done yet

• extra detections - means that psps has extra detections compared to what we expect from ipptopsps
• missing detections - means that psps has fewer detections compared to what we expect from ipptopsps

So far, MD02/MD03/MD04/MD06 are consistent (for the Stack/Detection tables)

MD01, MD05 has problems with Detections (missing)

MD07 has problems with Stack Detections and Detections (duplicates)

MD07 has problems with Stack Detections