Changes between Version 16 and Version 17 of PS1_IPP_Czarlog_20190527

Timestamp:

May 22, 2019, 5:17:03 PM (7 years ago)

Author:

fairlamb

Comment:

--

PS1_IPP_Czarlog_20190527

@@ -179 +179 @@
 The above gets the path_base for visit pairs (objects) and then finds the smf file to get the position data.
 
+
+
+We want to investigate the boresite info to assess how poor the PS2 pointing is. Things we should consider are:
+ - alt az position
+ - time of observation
+
+The query: 
+{{{
+mysql -hippdb08 -uX -pX gpc2 -B -e ' select path_base,exp_name,object,comment from camRun join camProcessedExp using (cam_id) join chipRun using (chip_id) join rawExp using (exp_id) where dateobs like "2019-05-22%" and ( comment like "%visit 1" or comment like "%visit 2" ) order by object; ' | grep -v path_base | awk '{print "neb-ls -p "$1".smf"}' | tcsh | fields OBJECT AST_R0 AST_D0
+}}}
+
+This provides us with pairs of visits (one on each line), so that we can compare the RA and DEC positions. 
+
+TODO: Just grab all those pointings, and plot them to see how they change across the night, for each night (just an MJD would be simple too)
+
+
+
+'''warp_stack_diff macro''' by JRF
+
+At the moment, the warpstack diffs processed by PS2 are not the best quality thanks to the poor masking and general unknown qualities of PS2. Coupled with this is the fact that the WWdiff and WSdiff processing is all done by stdscience for PS2 (unlike PS1 where it is separate). This means that stdscience could get clogged up with WSdiffs that keep failing and not process the WWdiffs as fast (the latter are critical for MOPS).
+
+What we plan to do is limit the time when warpstacks are run. This is determined by a macro living in file:
+~ippps2/stdscience/input 
+
+Looking at the ippitc version, there are the following macros:
+{{{
+## to change set the time UT
+macro warpstack.trange.all
+  task survey.warpstack.diff
+     trange -reset
+  end
+end
+
+macro warpstack.trange.day
+  task survey.warpstack.diff
+    trange -reset
+    trange        17:00:00 23:59:00
+    trange        00:00:00 05:00:00
+  end
+end
+}}}
+One for setting WS to be active all day, the other to only be during the daytime (note that the times are UTC time).
+
+These can be called during the setup, under the 'macro setup' section at the top of the input file, to automatically set when they should be used:
+{{{
+  #blah blah stuff
+
+  ## modify the trange for wsdiff:
+  ## trange.all -- run day and night; trange.day -- run only in the day
+  warpstack.trange.all
+  #warpstack.trange.day
+}}}
+You can only use one at a time (using more will overwrite the other). The above example shows the case where we are observing all the time.
+
+The macros are already in the input file for ippps2, but if they were not we could just copy the macros into the file and add the calling of them to the setup macro.
+
+The most important part is that we commented out 'warpstack.trange.all', and uncommented 'warpstack.trange.day' in the setup. Then WSdiffs will only be queued during the day.
+
+
+
+'''Euclid twilight observations''' by JRF
+
+Angie has fixed the Euclid comment strings to be more consistent with other observations, for example the before and after:
+{{{
+| BRIGHT   |      120 | Euclid twilight M8592 y; pos 10h offset from ps1_19_4388 Day 17 | 
+| BRIGHT   |      120 | Euclid.twilight.E8621.10a.z ps1_19_5863 Day 0                   | 
+}}}
+essentially the chunk names are now similar to the quad names used by MOPS and other parts of our processing.
+
+However, this means that they are no longer being picked up by the nightly_science processing. So we shall modify the nighty_science.config to fix this.
+
+nightly_science.config - contains the config for 
+nightly_science.pro - loads it
+nightly_science.pl - runs it all
+
+The config for PS1 lives here:
+{{{
+~ippitc/src/ipp-20170121/ippconfig/recipes/nightly_science.config
+}}}
+
+The config for handling the Euclid, bright, z-band observations during full moon has 'Euclid.r%' in their comments, while the ones in our case belong to different settings where it is looking for 'Euclid ' - note the space.
+
+The config looks like this:
+{{{
+TARGETS METADATA
+  NAME         STR Bright
+  DISTRIBUTION STR NODIST
+  TESS         STR RINGS.V3
+  OBSMODE      STR BRIGHT
+  STACKABLE   BOOL FALSE
+  DIFFABLE    BOOL TRUE
+  OFFNIGHT_DIFFS BOOL FALSE
+  DESPERATE_DIFFS BOOL TRUE
+  REDUCTION STR EUCLID_DEFAULT
+  COMMENT      STR  Euclid.r%
+#  DIFF          S16  7
+END
+TARGETS METADATA
+  NAME         STR BrightTwi
+  DISTRIBUTION STR NODIST
+  TESS         STR RINGS.V3
+  OBSMODE      STR BRIGHT
+  STACKABLE   BOOL FALSE
+  DIFFABLE    BOOL FALSE
+  OFFNIGHT_DIFFS BOOL FALSE
+  DESPERATE_DIFFS BOOL FALSE
+  REDUCTION STR EUCLID_DEFAULT
+  COMMENT      STR  Euclid 
+END
+}}}
+
+So, now we can change the config to handle the new comment format:
+{{{
+TARGETS METADATA
+  NAME         STR BrightTwi
+  DISTRIBUTION STR NODIST
+  TESS         STR RINGS.V3
+  OBSMODE      STR BRIGHT
+  STACKABLE   BOOL FALSE
+  DIFFABLE    BOOL FALSE
+  OFFNIGHT_DIFFS BOOL FALSE
+  DESPERATE_DIFFS BOOL FALSE
+  REDUCTION STR EUCLID_DEFAULT
+  COMMENT      STR  Euclid.twi%
+END
+}}}
+
+Save this, and let's check that the built version of the config file is similar (bar this change). The reason to check is that we don't want to erase any genuine changes in the build version -- remember this is a little cheeky just putting the file there, but it's because of how tied the config is to the build.
+
+The build file is here:
+{{{
+~ippitc/psconfig/ipp-20170121.lin64/share/ippconfig/recipes/nightly_science.config
+}}}
+
+Diff them:
+{{{
+diff nightly_science.config ~/src/ipp-20170121/ippconfig/recipes/nightly_science.config
+}}}
+
+If the only change is they one you're making great. In this case it was, so copy the src version across to the build version.
+
+
+''' queuing Euclid twilight observations ''' by JRF
+
+We may have changed the config to recognise the new Euclid format, but there have already been observations taken under the new format that have not been processed. Therefore, we need to send them to processing.
+
+To do this we are going to do something similar to running the test quad:
+
+An example of the test quad (as a reminder):
+{{{
+chiptool -dbname gpc2 -definebyquery -set_label MOPS.ps2testquad3.20190516ops -set_workdir neb://@HOST@.0/gpc2/MOPS.ps2testquad3.20190516ops -set_data_group MOPS.ps2testquad3.20190516ops -set_dist_group NULL -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction SWEETSPOT -simple -exp_name o8507h0371o -pretend
+}}}
+
+For processing these exposures the label, work_dir, data_group, dist_group, and reduction need to be changed (and expname of course).
+{{{
+chiptool -dbname gpc1 -definebyquery -set_label BrightTwi.nightlyscience -set_workdir neb://@HOST@.0/gpc1/BrightTwi.nt/2019/05/17/ -set_data_group BrightTwi.20190518 -set_dist_group NODIST -set_tess_id RINGS.V3 -set_end_stage warp -set_reduction EUCLID_DEFAULT -simple -exp_name  -pretend
+}}}
+
+So previously, if we look at something that has been done:
+
+select * from chipRun WHERE data_group LIKE "BrightTwi.201905%"; 
+
+TODO: script this up
+
+
+''' wait state cleaning ''' by JRF
+
+Have a look to see what is still waiting:
+{{{
+mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT waitct,data_group,label from (SELECT count(*) as waitct,data_group,label FROM chipRun WHERE state like "wait"  GROUP BY data_group) as s ORDER by waitct ASC;'
+}}}
+
+Choose a data group, and chuck it into the grinder:
+{{{
+mysql -hscidbm -uX -pX gpc1 -B -e 'SELECT chip_id,data_group,label FROM chipRun WHERE state like "wait" and data_group like "LAP.ThreePi.2014%" and label not like "update.LAP.PV3" LIMIT 5000;' | grep -v data_group | awk '{print "chiptool -dbname gpc1 -updaterun -set_state goto_cleaned -set_label goto_cleaned -chip_id  "$1}' | tcsh 
+}}}
+Just replace the data_group with whichever you want to clean. 
+Note that this one is limited to 5000 so as not to overload the system and that the label is not "update.LAP.PV3" so that we do not mess with the updates we're doing too.
+
+
+Have a check that they're doing ok:
+{{{
+    SELECT count(*), label FROM chipRun where state LIKE "goto_cleaned" GROUP BY label;
+}}}
+
+
 === Thursday : 2019.05.23 ===