[vsnet-grb 149649] New GCN Config file format and New Filtering Functions
Bacodine
vxw at capella2.gsfc.nasa.gov
Mon May 23 23:22:24 JST 2011
TO: All GCN Notice recipients
RE: New Sites Configuration method and new Filter Functions
DT: 23-May-11
INTRO:
A new format for the "sites.cfg" file has been adopted.
And new filtering functions have been added.
MOTIVATION FOR THE NEW FILE FORMAT:
There are several motivations for converting from the old fixed-format (hex numbers)
to the new free-format (english style) method:
1) The old config file was getting unwieldy -- too many columns;
and the new features/functions and new Notice types required more columns
that would have made it even more unwieldy.
2) This new "english style" format is easier to read.
3) Makes it easier for me to maintain/manage/daily_operations.
4) Allows Notice type-by-type filtering parameters: Error, TimeDelay, Intensity, etc;
which makes the Intensity filter work again because it is now able to handle
the variations in units and dynamic_ranges from notice type to notice type.
5) Allows for Notice type-by-type attachment specification.
6) Allows many other filtering functions (Cel/Gal/Ecl coordinates, Sun, Moon, etc).
CONFIGURATION FILE FORMAT CHANGE:
The switch is from the fixed-column hex-number format
with the entire configuration for a given site on a single line
to an english-style free-form "token value" format.
Examples of the old and new formats:
Old:
SBmail2 -77.45 38.85 -1 360.1 999.9 2C00 0002 3Aef e308 0000 0000 0 0 0 0 0 0 0 0 0 0 ALL NONE LMAIL scott at milkyway.gsfc.nasa.gov D C ; 25jun05 08jan11
New:
BEGIN_SITE
# Site_administrative items:
SITE_NAME SBmail2
DATES 25jun05 08jan11
SITE_LON_LAT -77.45 38.85
DIST_METHOD LMAIL
DIST_ADDRESS scott at milkyway.gsfc.nasa.gov
POC_NAME Scott Barthelmy
POC_ADDRESS scott at milkyway.gsfc.nasa.gov
DAILY_REPORT_ENABLE 0
DAILY_REPORT_ADDRESS scott at milkyway.gsfc.nasa.gov
# Site-level filter items:
TRIGGER_ID 0
INTENSITY -1.0
ERROR 360.1000
DELAY 999.9
SIGNIFICANCE 0.00
CONF_LEVEL 0.00
OBSERVABILITY ALL
ATTACHMENT NONE
SIMBAD_NED 0
# Type-level filter items:
INTEGRAL_WEAK
KONUS_LC
SWIFT_BAT_GRB_ALERT
SWIFT_BAT_GRB_POS trig_id=0
SWIFT_BAT_GRB_NACK_POS
SWIFT_BAT_TRANS
SWIFT_BAT_GRB_LC trig_id=0
SWIFT_FOM_2OBS trig_id=0
SWIFT_SC_2SLEW trig_id=0
SWIFT_XRT_POS
SWIFT_XRT_NACK_POS
SWIFT_XRT_IMAGE
SWIFT_XRT_PROC_IMAGE
SWIFT_UVOT_POS inten=99
SWIFT_UVOT_NACK_POS
SWIFT_UVOT_IMAGE
SWIFT_UVOT_PROC_IMAGE
SWIFT_UVOT_SRCLIST
SWIFT_UVOT_PROC_SRCLIST
SWIFT_TOO_FOM
SWIFT_TOO_SLEW
END_SITE
NEW FORMAT CONTENT:
In the example above, the configuration information for a site is divided up
into 3 parts: (a) the Site_adminstrative items, (b) the Site-level (global)
filtering speciications, and (c) the type-by-type-level filtering specifications.
For parts "a" and "b", the "token value" format provides self-documatation,
but for more detail, please refer to: http://gcn.gsfc.nasa.gov/tech_describe.html#tc27
Here is a template example of the part "c" Type-specific filtering format and syntax:
SWIFT_BAT_GRB_POS [inten=X] [error=Y] [delay=Z] [signif=S] [conf=P] [obs=all/vis/night/custom/etc] [trig_id=1/0] [sim_ned=1/0] [attach=sss]
The square brackets ([]) indicate that these are optional arguments;
the square brackets are not part of the actual invocation (eg "SWIFT_BAT_GRB_POS inten=500").
NEW FILTERING DESCRIPTION:
In the past, to receive a given notice it had to pass all 6 filter rules
of your configuration:
1) you had to have that notice type enabled AND
2) the position uncertainty had to be less than your error limit AND
3) it it had to be less than the time delay from the original trigger_time AND
4) the intensity measure of the event had to be greater than your intensity threshold AND
5) that particular trigger event had to be identified as the event type in question
if you had Trigger_ID enabled AND
6) the RA,Dec location on the sky had to pass your observability specification.
(These filter criteria stay exactly the same from the old way to the new way.)
The 10 additional filter rules add to the filtering process
making a 16-foled logical-AND requirement.
In the config-file format conversion from old to new,
the "wide open" default values for each of the new filter rules
has been specified for each site, so that they do not restrict
what you receive (e.g. the Sun_Distance is set to 0 degrees;
you get everything that is farther than 0 deg from the Sun).
It will be up to you to request changes to these new filter-rule values
to provide actual filtering operation for your configuration entry.
(And of course, the values for the original 6 filtering were copied from old to new.)
The old system of filtering had 6 dimensions:
Notice Type,
Error Size,
Time Delay,
Intensity (applicable to only the 4 (now obsolete) CGRO_BATSE-derived types),
Trigger_ID (applicable to Swift-BAT_Pos and the obsolete BATSE types),
Observability (All, Visible, Night, <custom>).
The new system of filtering has 16 dimensions
(the same 6 old dimensions plus 10 new dimensions):
Notice Type (defaults is disabled),
Error Size (default is 360 deg),
Time Delay (default is infinity hours),
Intensity (and now available for all types that have an "intensity" field; it is units specific),
Significance (default is 0.0),
Confidence (applicable to a small subset of types) (default is 0%),
Trigger_ID (applicable to a small subset of types) (default is 0),
Sun_distance (deg) (default is 0),
Sun_hour_angle_dist (hours) (default is 0),
Moon_distance (deg) (default is 0),
Moon_Illumination (%) (default is 100),
Celestial coords (RA,Dec ranges) (default is 0-360,-90-+90),
Galactic coords (Lon,Lat ranges) (default is 0-360,-90-+90),
Ecliptic coords (Lon,Lat ranges) (default is 0-360,-90-+90),
Time-of-day window (default is 0.0-24.0),
Observability (All, Visible, Night, <custom>) (default is All).
The meaning of each filtering parameters are:
Notice_type: The notice type has to be enabled to be received.
Error_size: The location error has to be less than the specified value.
Time_delay: The time delay has to be less than the specified value.
Intensity: The intensity has to be greater than the specified value.
Significance: The significance [sigma] has to be greater than the specified value.
Confidence: The confidence level [%] has to be greater than the specified value.
Trigger_ID: The trigger identity has to be what the TypeName indicates (ie no false positives).
Observability: The observability has to satisfy your All, Visible, Night, <custom> filter.
Sun_distance: The distance (in deg) to the Sun has to be greater than the specified value.
Sun_hour_angle_dist: The delta_RA hour_angle (lead or lag) has to be greater than the specified value.
Moon_distance: The distance (in deg) to the Moon has to be greater than the specified value.
Moon_illumination: The Moon illumination has to be less than the specified value.
Celestial_coords: The location has to be within the specified RA,Dec ranges.
Galactic_coords: The location has to be within the specified Gal_Lon,Lat ranges.
Ecliptic_coords: The location has to be within the specified Ecl_Lon,Lat ranges.
Time-of-day_window: The notice time has to be within the specified time window (UT).
And all of these can be reversed (negated). The "!" character
(as a suffix to the Token) is used to "reverse" the meaning of the filter range.
Example: The normal way to exclude notices during bright Moon time:
MOON_ILLUM 50
This will block all notices when the Moon is brighter than 50% illumination.
But if you want notices when the Moon IS bright
(eg. you have bright-time on a telescope to make near-IR observations):
MOON_ILLUM! 50
This will pass notices when the Moon is brighter than 50% illumination
(and block everything when the Moon is fainter than 50% illumination).
Other Examples:
If you wanted events ONLY on the Galactic Plane, then
GAL_LAT_LIM -20.0 +20.0
or if you want everything off the Plane (i.e. nothing IN the Plane)
GAL_LAT_LIM! -20.0 +20.0
If you only want items in the Galactic Bulge:
GAL_LON_LIM 340.0 +20.0
GAL_LAT_LIM -20.0 +20.0
If you have a northern hemisphere telescope, you can restrict
the Declination to what you can observe:
CEL_DEC_LIM -30 +90
If you only want items in the LMC:
CEL_RA_LIM 75.4 86.5
CEL_DEC_LIM -74 -64
The table below lists all the filter functions (old and new ways),
and it lists which functions are Global (i.e. applies to all
notice types) and which are Type-specific (i.e. can be specified
for each of the notice types). Type-specific override any
Global specifications. (The original 6 filter functions were only Global
in the old way.)
FILTER PARAM | OLD WAY(1) | NEW WAY(1)
| | Global | Type-Specific(4)
===============================================================================
Notice_type | X | X(3) | X
Error_size | X | X[!] | X
Time_delay | X | X[!] | X
Intensity | X(2) | X[!] | X
Significance | | X[!] | X
Confidence | | X[!] | X
Trigger_ID | X | X(3) | X
Observability | X | X(3) | X
| | |
Sun_distance (deg) | | X[!] |
Sun_hour_angle_dist (hr) | | X[!] |
Moon_distance (deg) | | X[!] |
Moon_illumination (%) | | X[!] |
| | |
Celestial coords (RA,Dec ranges) | | X[!] |
Galactic coords (Lon,Lat ranges) | | X[!] |
Ecliptic coords (Lon,Lat ranges) | | X[!] |
Time-of-day Window | | X[!] |
Table Notes:
(1) An "X" indicates that the given function is implemented in the Old or New ways.
Note that the first 8 filter rules are applicable at the global level
(i.e. to all Notice Types) and at the Type-by-Type level.
The next 8 filter rules (new way) are only applicable at the global level.
If you want different, say, Gal Lon,Lat values for different Notice Types,
then you will have to create two entries with the desired Types in each entry
with the different Gal Lon,Lat ranges.
(2) Intensity filtering was applicable to only the BATSE-based notice types (Old way).
Now it is applicable to all notice types that contain an intensity field;
and since it can be specified on a Type-by-Type basis, the different units and
dynamic ranges of intensities of each Type can be handled appropriately.
(3) There is no negation operation for the global Source_Type, Trigger_ID, and Observability.
If you don't want, say, SWIFT_BAT_POS, then don't enable it.
There is almost no practical need for the negation of Trigger_ID.
And it makes no practical sense to implement a "not visible" observability filter.
Negation capability on a filter function is indicated by the "[!]".
(4) There is no negation operation for the Type-specific filters.
The Type-specific specifications override the Global specifications. Example:
SIMBAD_NED 1
...
AGILE_GRB_WAKEUP_POS sim_ned=0
means you will get SIMBAD_NED notices for all the Position-based notice types
(that you have enabled), except for AGILE_GRB_WAKEUP_POS for which you have
specifically disabled its SIMBAD_NED notice.
Not all filtering functions are applicable to all notice types.
The table in http://gcn.gsfc.nasa.gov/gcn/filtering.html
shows which filters are active on which notice types. For notice types
for which the filter function is not defined, there is no filtering done
(no site-global, nor type-specific for that specific filter parameter);
.e. that particular notice always "passes" the test no matter what
is specified in the site's configuration.
There are some generalities and some uniques:
a) All intensity-containing notice types have intensity filtering. This is new;
the old way only had intensity filtering on BATSE_POS. This was because there
was only a global intensity threshold specification, and now there are
different instruments (each with their own threshold value and units)
a single number is not appropriate.
b) Significance filtering is somewhat different than intensity filtering.
It works in signal-to-noise sigma space (not counts or other raw units space).
For the notice types that provide only significance (not intensity),
GCN copies the significance value into the intensity slot in the notice packet.
c) Currently, there are only 2 notice types that have a Confidence Level
in their data packet: FERMI_GBM_FLT_POS and FERMI_GBM_TRANS.
d) Currently, there are only 4 notice types that have a Trigger_ID
in their data packet: SWIFT_BAT_GRB_POS, SWIFT_BAT_LIGHTCURVE, SWIFT_FOM_2OBS,
and SWIFT_SC_2SLEW.
RELATED FUNCTIONALITY:
1) The attachment_format specification is now also available
at the notice type-by-type level.
Example: if you want JPEG format for the XRT_BAT_LIGHTCURVE and FITS for the UVOT_IMAGE,
you can now do that:
SWIFT_BAT_LIGHTCURVE attach=jpeg
SWIFT_UVOT_IMAGE attach=fits
2) The SIMBAD_NED search notices are now also specifiable at the type-by-type level.
TESTING:
I have been running the new format & filtering method in parallel
with the old format and filtering method for about 12 months
with continuous crosschecking on a site-by-site notice-by-notice basis
and there have been no differences for the last 5 months
in the filtering/distribution decisions.
ACTION TO YOU:
1a) If you are happy with your current configuration (i.e. you do not want any
of the new filtering functions) then there is nothing to do.
(Your current configuration has been converted to the new format and
is appended below for your convenience.)
1b) If you do want any of these new filter functions or features,
go to the Config_builder web page http://gcn.gsfc.nasa.gov/gcn/config_builder.html ,
go to selection #2 (i.e. "Modify" a config),
type in your "sitename",
click "continue", and
change your configuration (click/unclick Notice radio buttons,
fill in filtering values, etc).
(I suggest you wait on making any Type-specific filtering changes
for Intensity, Significance, Confidence, or Trigger_ID. I have yet
to do the analysis (on past notices) to show the distributions
of those quantities vs the true-/false-positive_percentage.
Without that information, these Type-specific filters are of little value,
or could even be detrimental.)
2) Fill in the:
POC_NAME your_name_here
POC_ADDRESS your at address.goes.here
All sites are strongly encouraged to fill in the POC_NAME and POC_ADDRESS.
(For some sites I have filled these in based on recent email communications,
but I may be wrong, so please check/correct.)
3) Of course, you can always take this opportunity to request other changes
in your configuration. Your current configuration is listed below.
A reminder that MAXI_UNKNOWN/_KNOWN, INTEGRAL_WEAK, & BAT_MONITOR
have been aded to the list of available notice types in the last year.
Sincerely,
Scott
Scott Barthelmy NASA-GSFC, Code 661, Greenbelt, MD 20771
PHONE: 301-286-3106 (office)
PHONE: 301-346-3733 (cell)
FAX: 301-286-0677 (1st choice, -1682 2nd choice)
EMAIL: scott at lheamail.gsfc.nasa.gov
PAGER: 3013463733 at cingularme.com
WEB: http://gcn.gsfc.nasa.gov/gcn
//////////////////////// YOUR CURRENT CONFIGURATION ////////////////////////////
BEGIN_SITE
SITE_NAME VSNET
DATES 25jun05 23oct07
SITE_LON_LAT 135.75 35.00
DIST_METHOD EMAIL
DIST_ADDRESS vsnet-grb at ooruri.kusastro.kyoto-u.ac.jp
POC_NAME your_name_here
POC_ADDRESS your at address.goes.here
DAILY_REPORT_ENABLE 0
DAILY_REPORT_ADDRESS your at address.goes.here
# Site-specific items:
TRIGGER_ID 0
INTENSITY -1.0
ERROR 360.1000
DELAY 999.9
SIGNIFICANCE 0.00
CONF_LEVEL 0.00
OBSERVABILITY ALL
ATTACHMENT NONE
SIMBAD_NED 0
# Type-specific items:
IPN_SEG
IPN_POS
INTEGRAL_SPIACS
INTEGRAL_WAKEUP
INTEGRAL_REFINED
INTEGRAL_OFFLINE
KONUS_LC
SWIFT_BAT_GRB_ALERT
SWIFT_BAT_GRB_POS trig_id=0
SWIFT_BAT_GRB_NACK_POS
SWIFT_BAT_GRB_LC trig_id=0
SWIFT_FOM_2OBS trig_id=0
SWIFT_SC_2SLEW trig_id=0
SWIFT_XRT_POS
SWIFT_XRT_NACK_POS
SWIFT_XRT_IMAGE
SWIFT_XRT_PROC_IMAGE
SWIFT_UVOT_POS inten=99
SWIFT_UVOT_IMAGE
SWIFT_UVOT_PROC_IMAGE
SWIFT_UVOT_SRCLIST
SWIFT_UVOT_PROC_SRCLIST
SWIFT_POINTDIR
SWIFT_TOO_FOM
SWIFT_TOO_SLEW
END_SITE
More information about the vsnet-grb
mailing list