specprodDB API 

class specprodDB.coeff.ZtilePatch(*args: Any, **kwargs: Any)[source]

Table for patching Ztile table.

classmethod convert(data, tileid=None, night=None, row_index=None)[source]

Convert data into ORM objects ready for loading.

Parameters:

data (Table) – Data table to convert.
tileid (int, optional) – Tile ID number. If not set, it will be obtained from data.
night (int, optional) – Night number. This is loaded into the firstnight column. If not set, it will be obtained from data.
row_index (numpy.ndarray, optional) – Only convert the rows indexed by row_index. If not specified, convert all rows.

Returns:

A list of ORM objects.

Return type:

specprodDB.coeff.copy_columns(catalog, rowfilter, catalog_type)[source]

Copy columns from catalog into a new table.

Parameters:

catalog (Table) – Input data table.
rowfilter (callable) – A function that returns the subset of rows to include from catalog.
catalog_type (str) – Indicates the set of columns contained in catalog.

Returns:

A new Table with the desired columns.

Return type:

specprodDB.coeff.get_options(description='Extract coefficient columns to create a coeff patch table.')[source]

Parse command-line options.

Parameters:: description (str, optional) – Override the description in the command-line help.
Returns:: The parsed options.
Return type:: argparse.Namespace

specprodDB.coeff.main()[source]

Entry point for command-line script.

Returns:: An integer suitable for passing to sys.exit().
Return type:: int

specprodDB.load

Code for loading a spectroscopic production database. This includes both targeting and redshift data.

class specprodDB.load.Base(*args: Any, **kwargs: Any)[source]: SQLAlchemy 2.0 replacement for Base = declarative_base().

class specprodDB.load.Exposure(*args: Any, **kwargs: Any)[source]

Representation of the EXPOSURES HDU in the exposures file.

classmethod convert(data, row_index=None)[source]

Convert data into ORM objects ready for loading.

Parameters:

data (Table) – Data table to convert.
row_index (numpy.ndarray, optional) – Only convert the rows indexed by row_index. If not specified, convert all rows.

Returns:

A list of ORM objects.

Return type:

class specprodDB.load.Fiberassign(*args: Any, **kwargs: Any)[source]

Representation of the FIBERASSIGN table in a fiberassign file.

Notes

Targets are assigned to a location. A location happens to correspond to a fiber, but this correspondence could change over time, and therefore should not be assumed to be a rigid 1:1 mapping.
PLATE_RA, PLATE_DEC are sometimes missing. These can be copies of TARGET_RA, TARGET_DEC, but in principle they could be different if chromatic offsets in targeting positions were ever implemented.

classmethod convert(data, tileid=None, row_index=None)[source]

Convert data into ORM objects ready for loading.

Parameters:

data (Table) – Data table to convert.
tileid (int, optional) – Tile ID number. If not set, it will be obtained from data.
row_index (numpy.ndarray, optional) – Only convert the rows indexed by row_index. If not specified, convert all rows.

Returns:

A list of ORM objects.

Return type:

Raises:

KeyError – If tileid is not set and could not be obtained from data.

class specprodDB.load.Frame(*args: Any, **kwargs: Any)[source]

Representation of the FRAMES HDU in the exposures file.

Notes

The column frameid is a combination of expid and the camera name:

frameid = 100*expid + cameraid(camera)

where cameraid() is specprodDB.util.cameraid().

classmethod convert(data, row_index=None)[source]

Convert data into ORM objects ready for loading.

Parameters:

data (Table) – Data table to convert.
row_index (numpy.ndarray, optional) – Only convert the rows indexed by row_index. If not specified, convert all rows.

Returns:

A list of ORM objects.

Return type:

class specprodDB.load.Photometry(*args: Any, **kwargs: Any)[source]

Contains only photometric quantities associated with a TARGETID.

This table is deliberately designed so that TARGETID can serve as a primary key. Any quantities created or modified by desitarget are defined in the Target class.

However we avoid the use of the term “tractor” for this table, because not every target will have tractor photometry,

Notes

The various LC (light curve) columns, which are vector-valued, are not yet implemented.

classmethod convert(data, row_index=None)[source]

Convert data into ORM objects ready for loading.

Parameters:

data (Table) – Data table to convert.
row_index (numpy.ndarray, optional) – Only convert the rows indexed by row_index. If not specified, convert all rows.

Returns:

A list of ORM objects.

Return type:

class specprodDB.load.Potential(*args: Any, **kwargs: Any)[source]

Representation of the POTENTIAL_ASSIGNMENTS table in a fiberassign file.

classmethod convert(data, tileid=None, row_index=None)[source]

Convert data into ORM objects ready for loading.

Parameters:

data (Table) – Data table to convert.
tileid (int, optional) – Tile ID number. If not set, it will be obtained from data.
row_index (numpy.ndarray, optional) – Only convert the rows indexed by row_index. If not specified, convert all rows.

Returns:

A list of ORM objects.

Return type:

Raises:

KeyError – If tileid is not set and could not be obtained from data.

class specprodDB.load.SchemaMixin: Mixin class to allow schema name to be changed at runtime. Also automatically sets the table name.

class specprodDB.load.Target(*args: Any, **kwargs: Any)[source]

Representation of the pure-desitarget quantities in the TARGETPHOT table in the targetphot files.

classmethod convert(data, survey=None, tileid=None, row_index=None)[source]

Convert data into ORM objects ready for loading.

Parameters:

data (Table) – Data table to convert.
survey (str, optional) – Survey name. If not set, it will be obtained from data.
tileid (int, optional) – Tile ID number. If not set, it will be obtained from data.
row_index (numpy.ndarray, optional) – Only convert the rows indexed by row_index. If not specified, convert all rows.

Returns:

A list of ORM objects.

Return type:

Raises:

KeyError – If survey or tileid are not set and could not be obtained from data.

class specprodDB.load.Tile(*args: Any, **kwargs: Any)[source]

Representation of the tiles file.

Notes

Most of the data that are currently in the tiles file are derivable from the exposures table with much greater precision:

CREATE VIEW f5.tile AS SELECT tileid,
    -- SURVEY, FAPRGRM, FAFLAVOR?
    COUNT(*) AS nexp, SUM(exptime) AS exptime,
    MIN(tilera) AS tilera, MIN(tiledec) AS tiledec,
    SUM(efftime_etc) AS efftime_etc, SUM(efftime_spec) AS efftime_spec,
    SUM(efftime_gfa) AS efftime_gfa, MIN(goaltime) AS goaltime,
    -- OBSSTATUS?
    SUM(lrg_efftime_dark) AS lrg_efftime_dark,
    SUM(elg_efftime_dark) AS elg_efftime_dark,
    SUM(bgs_efftime_bright) AS bgs_efftime_bright,
    SUM(lya_efftime_dark) AS lya_efftime_dark,
    -- GOALTYPE?
    MIN(mintfrac) AS mintfrac, MAX(night) AS lastnight
FROM f5.exposure GROUP BY tileid;

However because of some unresolved discrepancies, we’ll just load the full tiles file for now.

classmethod convert(data, row_index=None)[source]

Convert data into ORM objects ready for loading.

Parameters:

data (Table) – Data table to convert.
row_index (numpy.ndarray, optional) – Only convert the rows indexed by row_index. If not specified, convert all rows.

Returns:

A list of ORM objects.

Return type:

class specprodDB.load.Version(*args: Any, **kwargs: Any)[source]

Store package version metadata.

classmethod convert(data, row_index=None)[source]

Convert the inputs into ORM objects ready for loading.

Parameters:

data (Table) – Data table to convert.
row_index (numpy.ndarray, optional) – Only convert the rows indexed by row_index. If not specified, convert all rows.

Returns:

A list of ORM objects.

Return type:

class specprodDB.load.Zpix(*args: Any, **kwargs: Any)[source]

Representation of the ZCATALOG table in zpix files.

classmethod convert(data, survey=None, program=None, row_index=None)[source]

Convert data into ORM objects ready for loading.

Parameters:

data (Table) – Data table to convert.
survey (str, optional) – Survey name. If not set, it will be obtained from data.
program (str, optional) – Program name. If not set, it will be obtained from data.
row_index (numpy.ndarray, optional) – Only convert the rows indexed by row_index. If not specified, convert all rows.

Returns:

A list of ORM objects.

Return type:

Raises:

KeyError – If survey or program are not set and the equivalent data are not available in data.

class specprodDB.load.Ztile(*args: Any, **kwargs: Any)[source]

Representation of the ZCATALOG table in ztile files.

classmethod convert(data, survey=None, program=None, tileid=None, night=None, row_index=None, spgrp='cumulative')[source]

Convert data into ORM objects ready for loading.

Parameters:

data (Table) – Data table to convert.
survey (str, optional) – Survey name. If not set, it will be obtained from data.
program (str, optional) – Program name. If not set, it will be obtained from data.
tileid (int, optional) – Tile ID number. If not set, it will be obtained from data.
night (int, optional) – Night number. This is loaded into the firstnight column. If not set, it will be obtained from data.
row_index (numpy.ndarray, optional) – Only convert the rows indexed by row_index. If not specified, convert all rows.
spgrp (str, optional) – Normally this will be set to the default value: ‘cumulative’.

Returns:

A list of ORM objects.

Return type:

Raises:

KeyError – If survey, program, tileid or night are not set and the equivalent data are not available in data.

Notes

If tileid is set, this method assumes data comes from one and only one tile.
The above has a secondary assumption that, at least for cumulative tile-based spectra, the first night is the same for all spectra.
night becomes firstnight, while spgrpval is equivalent to “lastnight” for cumulative tile-based spectra.

specprodDB.load.deduplicate_targetid(data)[source]

Find targetphot rows that are not already loaded into the Photometry table and resolve any duplicate TARGETID.

Parameters:: data (astropy.table.Table) – The initial data read from the file.
Returns:: An array of rows that are safe to load.
Return type:: numpy.ndarray

specprodDB.load.finitize(data, replacement_value=-9999.0)[source]

Convert NaN and other non-finite floating point values.

Parameters:

data (Table) – Data table to convert.
replacement_value (float, optional) – Replace NaN or other non-finite values with this value (default -9999.0).

Returns:

The input data modified in-place.

Return type:

specprodDB.load.get_options(description='Load redshift data into a specprod database.')[source]

Parse command-line options.

Parameters:: description (str, optional) – Override the description in the command-line help.
Returns:: The parsed options.
Return type:: argparse.Namespace

specprodDB.load.load_file(filepaths, tcls, hdu=1, row_filter=None, q3c=None, chunksize=50000, alternate_load=False)[source]

Load data file into the database, assuming that column names map to database column names with no surprises.

Parameters:

filepaths (str or list) – Full path to the data file or set of data files.
tcls (sqlalchemy.ext.declarative.api.DeclarativeMeta) – The table to load, represented by its class.
hdu (int or str, optional) – Read a data table from this HDU (default 1).
row_filter (callable, optional) – If set, apply this filter to the rows to be loaded. The function should return an array of indexes of “good” rows.
q3c (str, optional) – If set, create q3c index on the table, using the RA column named q3c.
chunksize (int, optional) – If set, load database chunksize rows at a time (default 50000).
alternate_load (bool, optional) – If True use an alternate loading scheme that may reduce memory use.

Returns:

The grand total of rows loaded.

Return type:

specprodDB.load.load_versions(photometry, redshift, release, specprod, tiles)[source]

Load version metadata.

The inputs to this function are normally specified in the specprod configuration file. Other necessary metadata are obtained at import time.

Parameters:

photometry (str) – Photometry catalog.
redshift (str) – Redshift catalog version.
release (str) – Data release, e.g. ‘edr’, ‘dr1’.
specprod (str) – The specprod version. Usually, but not always, the same as the schema name.
tiles (str) – The tiles (fiberassign file) version.

specprodDB.load.main()[source]

Entry point for command-line script.

Returns:: An integer suitable for passing to sys.exit().
Return type:: int

specprodDB.load.q3c_index(table, ra='ra')[source]

Create a q3c index on a table.

Parameters:

table (str) – Name of the table to index.
ra (str, optional) – If the RA, Dec columns are called something besides “ra” and “dec”, set its name. For example, ra='target_ra'.

specprodDB.load.setup_db(dbfile='specprod.db', hostname=None, username='desi_admin', schema=None, overwrite=False, public=False, verbose=False)[source]

Initialize the database connection.

Parameters:

dbfile (str, optional) – Name of a SQLite file for output (default specprod.db). If no path is specified in the file name, the current working directory will be used.
hostname (str, optional) – Name of a PostgreSQL server for output.
username (str, optional) – Username on a PostgreSQL server for database connection.
schema (str, optional) – Name of database schema that will contain output tables.
overwrite (bool, optional) – If True, overwrite any existing schema or table.
public (bool, optional) – If True, allow public access to the database or schema.
verbose (bool, optional) – If True, Print extra debugging information for SQL queries.

Returns:

True if the configured database is a PostgreSQL database.

Return type:

bool

Raises:

RuntimeError – If database connection details could not be found.

specprodDB.load.upsert(rows, do_nothing=False)[source]

Convert a list of ORM objects into an INSERT ... ON CONFLICT statement.

Parameters:

rows (list) – A list of ORM objects. All items should be the same type.
do_nothing (bool, optional) – If True, do not attempt to update existing rows.

Returns:

A specialzed INSERT statement ready for execution.

Return type:

Insert

specprodDB.patch

Patch top-level exposures and tiles summary files. This is meant to be applied to the daily specprod, but others could be patched, in principle.

specprodDB.patch.back_patch_inconsistent_values(patched)[source]

When the primary round of patching is done, copy some values back into the exposures and frames files.

Parameters:: patched (dict) – A dictionary containing tables for further patching.
Returns:: A tuple containing the back-patched exposures and frames tables. Not strictly necessary as this function will modify the tables in patched in-place.
Return type:: tuple

specprodDB.patch.get_data(options)[source]

Read in source and destination data.

:param argparse.Namespace: The parsed command-line options.

Returns:: A tuple containing two dictionaries, each containing three Table objects, plus some metadata.
Return type:: tuple

specprodDB.patch.get_options()[source]

Parse command-line options.

Returns:: The parsed options.
Return type:: argparse.Namespace

specprodDB.patch.main()[source]

Entry point for command-line script.

Returns:: An integer suitable for passing to sys.exit().
Return type:: int

specprodDB.patch.match_rows(left, right)[source]

Match rows in left to rows in right.

Parameters:

src (array-like) – The column to be matched. This could be “artificial”.
dst (array-like) – The column to be matched. This could be “artificial”.

Returns:

The row indexes of left and right that match.

Return type:

tuple

specprodDB.patch.patch_exposures(src_exposures, dst_exposures, first_night=None)[source]

Patch exposures data in dst_exposures with the data in src_exposures.

Parameters:

src_exposures (Table) – Source of exposures data.
dst_exposures (Table) – Data to be patched.
first_night (int, optional) – The earliest night of data that will ever be loaded. If not set, it will be derived from the first exposure in src_exposures

Returns:

A copy of dst_exposures with data replaced from src_exposures.

Return type:

specprodDB.patch.patch_exposures_efftime_spec(src_exposures, dst_exposures, dst_tiles)[source]

Patch exposures that have EFFTIME_SPEC == 0 where the corresponding tile has EFFTIME_SPEC > 0.

Parameters:

src_exposures (Table) – Source of exposures data.
dst_exposures (Table) – Data to be patched.
dst_tiles (Table) – Tiles data for comparison. Should not be modified.

Returns:

A copy of dst_exposures with data replaced from src_exposures.

Return type:

specprodDB.patch.patch_frames(src_frames, dst_frames)[source]

Patch frames data in dst_frames with the data in src_frames.

Parameters:

src_frames (Table) – Source of frames data.
dst_frames (Table) – Data to be patched.

Returns:

A copy of dst_frames with data replaced from src_frames.

Return type:

specprodDB.patch.patch_missing_frames_mjd(exposures, frames)[source]

Update MJD values in frames after exposures has been patched.

Parameters:

exposures (Table) – Patched exposures table.
frames (Table) – Patched frames table.

Returns:

An updated version of frames.

Return type:

specprodDB.patch.patch_tiles(src_tiles, dst_tiles, timestamp)[source]

Patch frames data in dst_tiles with the data in src_tiles.

Parameters:

src_tiles (Table) – Source of tiles data.
dst_tiles (Table) – Data to be patched.
timestamp (datetime.datetime) – Fill value for the UPDATED column.

Returns:

A copy of dst_tiles with data replaced from src_tiles.

Return type:

specprodDB.patch.zero_fill(data, label)[source]

Fill any masked values in data with zero.

Parameters:

data (Table) – A data table.
label (str) – A label to use in logging.

Returns:

The modified data table.

Return type:

specprodDB.tile

Code for loading one or more separate tiles into the spectroscopic production database.

Notes

Some of this code may be combined or otherwise refactored with specprodDB.load in the future.

specprodDB.tile.fiberassign_file(tileid)[source]

Find a fiberassign file associated with tileid.

Parameters:: tileid (int) – The Tile ID.
Returns:: The path to the fiberassign file corresponding to tileid.
Return type:: str

specprodDB.tile.get_options(description='Load data for one tile into a specprod database.')[source]

Parse command-line options.

Parameters:: description (str, optional) – Override the description in the command-line help.
Returns:: The parsed options.
Return type:: argparse.Namespace

specprodDB.tile.load_fiberassign(tile)[source]

Load the fiber assignments and potential assignments for tile.

Parameters:: tile (Tile) – The tile with fiber assignments to load.
Returns:: A tuple containing the lists of Fiberassign and Potential objects loaded.
Return type:: tuple

specprodDB.tile.load_photometry(photometry)[source]

Insert the data in photometry into the database.

Parameters:: photometry (Table) – A Table containing the photometry data.
Returns:: A list of Photometry objects loaded.
Return type:: list

specprodDB.tile.load_redshift(tile, spgrp='cumulative')[source]

Load redshift data associated with tile.

Parameters:

tile (Tile) – The tile with redshifts to load.
spgrp (str, optional) – The type of redshift data to load. Currently only ‘cumulative’ is supported.

Returns:

A list of Ztile objects loaded.

Return type:

Raises:

ValueError – If the value of spgrp is not supported.

specprodDB.tile.load_target(tile, target)[source]

Load the targeting data associated with tile.

Parameters:

tile (Tile) – The tile associated with target.
target (Table) – Effectively a list of TARGETID.

Returns:

A list of Target objects loaded.

Return type:

specprodDB.tile.load_targetphot(targetphot, loaded_photometry)[source]

Load the photometry, such as it is, for objects that do not have Tractor photometry.

Parameters:

targetphot (Table) – A Table containing the targeting data.
loaded_photometry (list) – A list of Photometry objects already loaded.

Returns:

A list of Photometry objects loaded.

Return type:

specprodDB.tile.main()[source]

Entry point for command-line script.

Returns:: An integer suitable for passing to sys.exit().
Return type:: int

specprodDB.tile.potential_photometry(tile, targets)[source]

Assemble a Table of targets that will be used to find photometric data.

targets is assumed to come from one tile that has not already been loaded. Any existing photometry already loaded will be excluded from the returned list.

Parameters:

tile (Tile) – The tile associated with targets.
targets (Table) – Effectively a list of TARGETID.

Returns:

A Table that will be the input to photometric search functions.

Return type:

specprodDB.tile.potential_targets(tileid)[source]

Find potential targets associated with tileid.

Sky targets are not returned.

Parameters:: tileid (int) – The Tile ID.
Returns:: A table containing potential target information.
Return type:: Table

specprodDB.tile.targetphot(catalog)[source]

Find the target data associated with the targets in catalog.

Parameters:: catalog (Table) – A list of objects.
Returns:: A Table containing the targeting data.
Return type:: Table

specprodDB.tile.tractorphot(catalog)[source]

Find the photometry data associated with the targets in catalog.

Parameters:: catalog (Table) – A list of objects.
Returns:: A Table containing the photometry data.
Return type:: Table

specprodDB.tile.update_primary()[source]: Update the primary classification after some number of tiles has been loaded.

specprodDB.tile.update_q3c()[source]: Update the q3c indexes after some number of tiles has been loaded.

specprodDB.util

Classes and functions for use by all database code.

specprodDB.util.cameraid(camera)[source]

Converts camera (e.g. ‘b0’) to an integer in a simple but ultimately arbitrary way.

Parameters:: camera (str) – Camera name.
Returns:: An arbitrary integer, though in the range [0, 29].
Return type:: int

specprodDB.util.checkgzip(filename)[source]

Check for existence of filename, with or without a .gz extension.

Parameters:: filename (str) – Filename to check.
Returns:: Path of existing file with or without .gz.
Return type:: str
Raises:: FileNotFoundError – If neither file type exists.

specprodDB.util.common_options(description)[source]

Define a set of common command-line options.

Individual command-line scripts will add additional options.

Parameters:: description (str) – Define the description in the command-line help.
Returns:: An argument parser to which further arguments may be added.
Return type:: ArgumentParser

specprodDB.util.convert_dateobs(timestamp, tzinfo=None)[source]

Convert a string timestamp into a datetime.datetime object.

Parameters:

timestamp (str) – Timestamp in string format.
tzinfo (datetime.tzinfo, optional) – If set, add time zone to the timestamp.

Returns:

The converted timestamp.

Return type:

datetime.datetime

specprodDB.util.decode_surveyid(surveyid)[source]

Converts surveyid to its corresponding name.

Parameters:: surveyid (int) – Survey number
Returns:: The name of the corresponding survey.
Return type:: str

specprodDB.util.decode_targetphotid(targetphotid)[source]

Convert id into its components.

Parameters:: targetphotid (int) – The 128-bit id.
Returns:: A tuple of targetid, tileid and survey.
Return type:: tuple

specprodDB.util.fiberassignid(targetid, tileid, location)[source]

Convert inputs into an arbitrary large integer.

Parameters:

targetid (int) – Standard TARGETID.
tileid (int) – Standard TILEID.
location (int) – Location on the tile.

Returns:

An arbitrary integer, which will be greater than \(2^64\) but less than \(2^128\).

Return type:

specprodDB.util.frameid(expid, camera)[source]

Converts the pair expid, camera into an arbitrary integer suitable for use as a primary key.

Parameters:

expid (int) – Exposure ID associated with the frame.
camera (str) – Camera name.

Returns:

An arbitrary integer.

Return type:

specprodDB.util.no_sky(catalog)[source]

Identify objects in catalog that are not sky targets.

Parameters:: catalog (Table) – Any Table containing a TARGETID column.
Returns:: The indexes of rows that are not sky targets.
Return type:: numpy.ndarray

specprodDB.util.parse_pgpass(hostname='specprod-db.desi.lbl.gov', username='desi_admin')[source]

Read a ~/.pgpass file.

Parameters:

hostname (str, optional) – Database hostname.
username (str, optional) – Database username.

Returns:

A string suitable for creating a SQLAlchemy database engine, or None if no matching data was found.

Return type:

str

specprodDB.util.programid(program)[source]

Converts program (e.g. ‘bright’) to an integer in a simple but ultimately arbitrary way.

Parameters:: program (str) – Program name.
Returns:: An arbitrary, small integer.
Return type:: int

specprodDB.util.spgrpid(spgrp)[source]

Converts spgrp (e.g. ‘cumulative’) to an integer in a simple but ultimately arbitrary way.

Parameters:: spgrp (str) – SPGRP name.
Returns:: An arbitrary, small integer.
Return type:: int

specprodDB.util.surveyid(survey)[source]

Converts survey (e.g. ‘main’) to an integer in a simple but ultimately arbitrary way.

Parameters:: survey (str) – Survey name.
Returns:: An arbitrary, small integer.
Return type:: int

specprodDB.util.targetphotid(targetid, tileid, survey)[source]

Convert inputs into an arbitrary large integer.

Parameters:

targetid (int) – Standard TARGETID.
tileid (int) – Standard TILEID.
survey (str) – Survey name.

Returns:

An arbitrary integer, which will be greater than \(2^64\) but less than \(2^128\).

Return type:

specprodDB.util.zpixid(targetid, survey, program)[source]

Convert inputs into an arbitrary large integer.

Parameters:

targetid (int) – Standard TARGETID.
survey (str) – Survey name.
program (str) – Program name.

Returns:

An arbitrary integer, which will be greater than \(2^64\) but less than \(2^128\).

Return type:

specprodDB.util.ztileid(targetid, spgrp, spgrpval, tileid)[source]

Convert inputs into an arbitrary large integer.

Parameters:

targetid (int) – Standard TARGETID.
spgrp (str) – Tile grouping.
spgrpval (int) – Id within spgrp.
tileid (int) – Standard TILEID.

Returns:

An arbitrary integer, which will be greater than \(2^64\) but less than \(2^128\).

Return type: