Dynamic Loading for daily Reductions

Daily Database Loading Requirements

  • The tiles-daily file should have some indications of which tiles need to be added, updated or deleted, i.e. INSERT, UPDATE, DELETE. Hopefully deletes will be rare.

    • The tiles-daily file is only appended-to, old entries are not necessarily updated. See desihub/desispec#2251 for more details.

    • There are many other potential issues with the desi_tsnr_afterburner script. The issue linked above has details.

  • Load redshifts from redrock files in tiles/cumulative, rather than from the zcatalog summary files. HEALPix redshifts are not and probably never will be generated in daily reductions.

  • Compute target and photometry data based on the objects being loaded rather than reading all at once. See further discussion below.

  • For further discussion see #13.

Possible Load Scenario

For individual tiles

  1. Read tiles-daily, find changes. Update daily.tile.

  2. Find corresponding exposures. Update daily.exposure, daily.frame.

  3. Find any new fiberassign files and obtain the list of new potentials (because potentials include observed targets).

  4. Obtain the targeting and tractor data for new potentials. Update daily.photometry and daily.target. Take care to check for existing entries.

  5. Make sure to perform the equivalent of the targetphot stage, i.e. fill in photometric data from targeting data.

  6. Update daily.fiberassign, daily.potential.

  7. Read corresponding tiles/cumulative redshift files. Update daily.ztile.

For initial load

  1. Load the exposures and tiles files.

  2. Compute “monolithic” redshift catalogs. These are needed as input to photometry.

  3. Create “monolithic” photometry and target catalogs.

  4. The load procedure then resembles loading a static specprod, just without healpix redshifts.

  5. Update the “primary” columns and any q3c indexes.

Automated Extraction of Targeting and Photometry

  • targetphot_onetile() in mpi-photometry appears to work for both targeted and potential objects.

  • tractorphot_onebrick() is similar.

  • Both are thin wrappers on desispec.io.photo.gather_(tractor|target)phot().

  • But the gather functions don’t necessarily add DESINAME or fix NaN values for PMRA or PMDEC. Make sure inputs and outputs match.

  • In the database, desiname is attached to the redshift tables, not the photometry or target tables. So it should be added at that stage, presumably.

Other Notes

  • Plan for how to support fuji+guadalupe combined analysis. May need to look into cross-schema views, or daughter tables that inherit from both schemas.

  • Anticipate loading afterburners and VACs into the database.

  • How do q3c indexes work with dynamic loading?