Public Member Functions | Public Attributes | Static Public Attributes | Private Member Functions | Private Attributes | List of all members
grepros.plugins.auto.postgres.PostgresSink Class Reference
Inheritance diagram for grepros.plugins.auto.postgres.PostgresSink:
Inheritance graph
[legend]

Public Member Functions

def __init__ (self, args=None, **kwargs)
 
def autodetect (cls, target)
 
def validate (self)
 
- Public Member Functions inherited from grepros.plugins.auto.dbbase.BaseDataSink
def close (self)
 
def close_output (self)
 
def emit (self, topic, msg, stamp=None, match=None, index=None)
 
- Public Member Functions inherited from grepros.outputs.Sink
def __enter__ (self)
 
def __exit__ (self, exc_type, exc_value, traceback)
 
def bind (self, source)
 
def emit_meta (self)
 
def flush (self)
 
def is_highlighting (self)
 
def thread_excepthook (self, text, exc)
 

Public Attributes

 valid
 
- Public Attributes inherited from grepros.plugins.auto.dbbase.BaseDataSink
 COMMIT_INTERVAL
 
 db
 Database connection. More...
 
- Public Attributes inherited from grepros.outputs.Sink
 args
 
 source
 inputs.Source instance bound to this sink More...
 
 valid
 Result of validate() More...
 

Static Public Attributes

string ENGINE = "Postgres"
 Database engine name. More...
 
int ID_SEQUENCE_STEP = 100
 Sequence length per table to reserve for inserted message IDs. More...
 
list MESSAGE_TYPE_BASECOLS
 Default columns for pkg/MsgType tables. More...
 
list MESSAGE_TYPE_NESTCOLS
 Additional default columns for pkg/MsgType tables with nesting output. More...
 
list MESSAGE_TYPE_TOPICCOLS
 Default topic-related columns for pkg/MsgType tables, as [(name, SQL type)]. More...
 
string SELECT_TYPE_COLUMNS
 SQL statement for selecting metainfo on pkg/MsgType table columns. More...
 
- Static Public Attributes inherited from grepros.plugins.auto.dbbase.BaseDataSink
int COMMIT_INTERVAL = 1000
 Number of emits between commits; 0 is autocommit. More...
 
 DEFAULT_ARGS = dict(META=False, WRITE_OPTIONS={}, VERBOSE=False)
 Constructor argument defaults. More...
 
 ENGINE = None
 Database engine name, overridden in subclasses. More...
 
list MESSAGE_TYPE_BASECOLS
 Default columns for pkg/MsgType tables. More...
 
list MESSAGE_TYPE_NESTCOLS
 Additional default columns for pkg/MsgType tables with nesting output. More...
 
list MESSAGE_TYPE_TOPICCOLS
 Default topic-related columns for pkg/MsgType tables. More...
 
- Static Public Attributes inherited from grepros.outputs.Sink
 DEFAULT_ARGS = dict(META=False)
 Constructor argument defaults. More...
 
tuple FILE_EXTENSIONS = ()
 Auto-detection file extensions for subclasses, as (".ext", ) More...
 
- Static Public Attributes inherited from grepros.plugins.auto.sqlbase.SqlMixin
 DEFAULT_ARGS = dict(META=False, WRITE_OPTIONS={}, MATCH_WRAPPER=None, VERBOSE=False)
 Constructor argument defaults. More...
 
string DEFAULT_DIALECT = "sqlite"
 Default SQL dialect used if dialect not specified. More...
 
dictionary DIALECTS
 Supported SQL dialects and options. More...
 
list KEYWORDS
 Words that need quoting if in name context, like table name. More...
 

Private Member Functions

def _connect (self)
 
def _execute_insert (self, sql, args)
 
def _executemany (self, sql, argses)
 
def _executescript (self, sql)
 
def _get_next_id (self, table)
 
def _init_db (self)
 
def _load_schema (self)
 
def _make_column_value (self, value, typename=None)
 
def _make_db_label (self)
 

Private Attributes

 _id_queue
 

Detailed Description

Writes messages to a Postgres database.

Output will have:
- table "topics", with topic and message type names
- table "types", with message type definitions

plus:
- table "pkg/MsgType" for each topic message type, with detailed fields,
  BYTEA fields for uint8[], array fields for scalar list attributes,
  and JSONB fields for lists of ROS messages;
  with foreign keys if nesting else subtype values as JSON dictionaries;
  plus underscore-prefixed fields for metadata, like `_topic` as the topic name.
  If not nesting, only topic message type tables are created.
- view "/full/topic/name" for each topic, selecting from the message type table

If a message type table already exists but for a type with a different MD5 hash,
the new table will have its MD5 hash appended to end, as "pkg/MsgType (hash)".

Definition at line 30 of file postgres.py.

Constructor & Destructor Documentation

◆ __init__()

def grepros.plugins.auto.postgres.PostgresSink.__init__ (   self,
  args = None,
**  kwargs 
)
@param   args                 arguments as namespace or dictionary, case-insensitive;
                      or a single item as the database connection string
@param   args.write           Postgres connection string like "postgresql://user@host/db"
@param   args.write_options   ```
                      {"commit-interval": transaction size (0 is autocommit),
                       "nesting": "array" to recursively insert arrays
                                  of nested types, or "all" for any nesting)}
                      ```
@param   args.meta            whether to emit metainfo
@param   args.verbose         whether to emit debug information
@param   kwargs               any and all arguments as keyword overrides, case-insensitive

Reimplemented from grepros.plugins.auto.dbbase.BaseDataSink.

Definition at line 79 of file postgres.py.

Member Function Documentation

◆ _connect()

def grepros.plugins.auto.postgres.PostgresSink._connect (   self)
private
Returns new database connection.

Reimplemented from grepros.plugins.auto.dbbase.BaseDataSink.

Definition at line 137 of file postgres.py.

◆ _execute_insert()

def grepros.plugins.auto.postgres.PostgresSink._execute_insert (   self,
  sql,
  args 
)
private
Executes INSERT statement, returns inserted ID.

Reimplemented from grepros.plugins.auto.dbbase.BaseDataSink.

Definition at line 143 of file postgres.py.

◆ _executemany()

def grepros.plugins.auto.postgres.PostgresSink._executemany (   self,
  sql,
  argses 
)
private
Executes SQL with all args sequences.

Reimplemented from grepros.plugins.auto.dbbase.BaseDataSink.

Definition at line 149 of file postgres.py.

◆ _executescript()

def grepros.plugins.auto.postgres.PostgresSink._executescript (   self,
  sql 
)
private
Executes SQL with one or more statements.

Reimplemented from grepros.plugins.auto.dbbase.BaseDataSink.

Definition at line 154 of file postgres.py.

◆ _get_next_id()

def grepros.plugins.auto.postgres.PostgresSink._get_next_id (   self,
  table 
)
private
Returns next cached ID value, re-populating empty cache from sequence.

Reimplemented from grepros.plugins.auto.dbbase.BaseDataSink.

Definition at line 159 of file postgres.py.

◆ _init_db()

def grepros.plugins.auto.postgres.PostgresSink._init_db (   self)
private
Opens the database file, and populates schema if not already existing.

Reimplemented from grepros.plugins.auto.dbbase.BaseDataSink.

Definition at line 116 of file postgres.py.

◆ _load_schema()

def grepros.plugins.auto.postgres.PostgresSink._load_schema (   self)
private
Populates instance attributes with schema metainfo.

Reimplemented from grepros.plugins.auto.dbbase.BaseDataSink.

Definition at line 124 of file postgres.py.

◆ _make_column_value()

def grepros.plugins.auto.postgres.PostgresSink._make_column_value (   self,
  value,
  typename = None 
)
private
Returns column value suitable for inserting to database.

Reimplemented from grepros.plugins.auto.sqlbase.SqlMixin.

Definition at line 172 of file postgres.py.

◆ _make_db_label()

def grepros.plugins.auto.postgres.PostgresSink._make_db_label (   self)
private
Returns formatted label for database.

Reimplemented from grepros.plugins.auto.dbbase.BaseDataSink.

Definition at line 203 of file postgres.py.

◆ autodetect()

def grepros.plugins.auto.postgres.PostgresSink.autodetect (   cls,
  target 
)
Returns true if target is recognizable as a Postgres connection string.

Reimplemented from grepros.outputs.Sink.

Definition at line 211 of file postgres.py.

◆ validate()

def grepros.plugins.auto.postgres.PostgresSink.validate (   self)
Returns whether Postgres driver is available,
and "commit-interval" and "nesting" in args.write_options have valid value, if any,
and database is connectable.

Reimplemented from grepros.plugins.auto.dbbase.BaseDataSink.

Definition at line 97 of file postgres.py.

Member Data Documentation

◆ _id_queue

grepros.plugins.auto.postgres.PostgresSink._id_queue
private

Definition at line 94 of file postgres.py.

◆ ENGINE

string grepros.plugins.auto.postgres.PostgresSink.ENGINE = "Postgres"
static

Database engine name.

Definition at line 52 of file postgres.py.

◆ ID_SEQUENCE_STEP

int grepros.plugins.auto.postgres.PostgresSink.ID_SEQUENCE_STEP = 100
static

Sequence length per table to reserve for inserted message IDs.

Definition at line 55 of file postgres.py.

◆ MESSAGE_TYPE_BASECOLS

list grepros.plugins.auto.postgres.PostgresSink.MESSAGE_TYPE_BASECOLS
static
Initial value:
= [("_dt", "TIMESTAMP"),
("_timestamp", "NUMERIC"),
("_id", "BIGSERIAL PRIMARY KEY"), ]

Default columns for pkg/MsgType tables.

Definition at line 71 of file postgres.py.

◆ MESSAGE_TYPE_NESTCOLS

list grepros.plugins.auto.postgres.PostgresSink.MESSAGE_TYPE_NESTCOLS
static
Initial value:
= [("_parent_type", "TEXT"),
("_parent_id", "BIGINT"), ]

Additional default columns for pkg/MsgType tables with nesting output.

Definition at line 75 of file postgres.py.

◆ MESSAGE_TYPE_TOPICCOLS

list grepros.plugins.auto.postgres.PostgresSink.MESSAGE_TYPE_TOPICCOLS
static
Initial value:
= [("_topic", "TEXT"),
("_topic_id", "BIGINT"), ]

Default topic-related columns for pkg/MsgType tables, as [(name, SQL type)].

Definition at line 68 of file postgres.py.

◆ SELECT_TYPE_COLUMNS

string grepros.plugins.auto.postgres.PostgresSink.SELECT_TYPE_COLUMNS
static
Initial value:
= """
SELECT c.table_name, c.column_name, c.data_type
FROM information_schema.columns c INNER JOIN information_schema.tables t
ON t.table_name = c.table_name
WHERE c.table_schema = current_schema() AND t.table_type = 'BASE TABLE' AND
c.table_name LIKE '%/%'
ORDER BY c.table_name, CAST(c.dtd_identifier AS INTEGER)
"""

SQL statement for selecting metainfo on pkg/MsgType table columns.

Definition at line 58 of file postgres.py.

◆ valid

grepros.plugins.auto.postgres.PostgresSink.valid

Definition at line 112 of file postgres.py.


The documentation for this class was generated from the following file:


grepros
Author(s): Erki Suurjaak
autogenerated on Sat Jan 6 2024 03:11:30