Python Classes

constants

constants.py

class apicrud.AESEncrypt(secret)

AES encryption for strings

Provides easier-to-use AES CBC encrypt/decrypt operations for strings

Parameters

secret (str) – passphrase (suggest at least 16 characters)

decrypt(enc)

decrypt an object

Parameters

enc (bytes) – encrypted object

Returns

decrypted string

Return type

str

encrypt(raw)

encrypt a string

Parameters

raw (str) – object to be encrypted

Returns

encrypted object

Return type

bytes

class apicrud.AESEncryptBin(secret)

AES encryption for binaries

Provides easier-to-use AES CBC encrypt/decrypt operations for byte objects

Parameters

secret (str) – passphrase (suggest at least 16 characters)

decrypt(enc)

decrypt an object

Parameters

enc (bytes) – encrypted object

Returns

decrypted bytes

Return type

bytes

encrypt(raw)

encrypt a byte object

Parameters

raw (bytes) – object to be encrypted

Returns

encrypted object

Return type

bytes

class apicrud.AccessControl(policy_file=None, model=None)

Role-based access control

Definitions:

  • principal: a user or role

  • membership: parent resource type for privacy sharing

  • model: database model name (e.g. Person)

  • resource: resource type (e.g. person)

  • rbac: role-based access control (defined in rbac.yaml)

  • role: a group name (e.g. admin or list-<id>-<level>)

  • privacy: sharing options as defined in rbac.yaml (e.g. secret [default], public, invitee, member, manager)

  • actions: crudlghij (create, read, update, del, list, guest/member, host/manager, invitee, join)

In rbac.yaml, define the RBAC policies for each principal/resource combination. That file will be parsed into a singleton variable upon initial startup. This implementation implements RBAC similar to that of kubernetes or AWS IAM, with the added capability of a simple privacy permission within each object (database record) which creates an implied ACL for read-only access by members of the object’s group.

Group names currently used are:

  • admin

  • user

  • pending (new-account confirmation)

  • pendingtotp

  • person

  • <resource>-<id>-<privacy>

These are defined in session_auth.py’s account_login() method.

One resource can be used to define user’s group memberships with a rbac.yaml entry in this form:

private_resources:
  • resource: list attr: list_id

A user who is a member of list “wildlife-lovers” which has an id x-12345678 would get an auth role “list-x-12345678-member”.

Parameters
  • policy_file (str) – name of the yaml definitions file

  • model (obj) – a model to be validated for permissions

apikey_create()

Generate an API key - a 41-byte string. First 8 characters (48 bits) are an access key ID prefix; last 32 characters (192 bits) are the secret key.

Returns: tuple

key ID (str) - public portion of key secret (str) - secret portion hashvalue (str) - hash value for database

static apikey_hash(secret)

Generate a hash value from the secret :param secret: secret key :type secret: str

apikey_verify(key_id, secret)

Verify an API key

Parameters
  • key_id (str) – the public key_id at time of generation

  • secret (str) – the unhashed secret

Returns: tuple

uid (str): User ID if valid scopes (list): list of scope IDs

load_rbac(filename)

Read RBAC default policies from rbac.yaml, process any string substitutions, and convert * for re.match()

Parameters

filename (str) – filename containing RBAC definitions

rbac_permissions(query=None, owner_uid=None, membership=None, id=None, privacy=None)

Evaluate an access request for self.auth roles of self.uid in self.resource against defined policies

Parameters
  • query (obj) – an existing record (takes precedence over owner_uid)

  • owner_uid (str) – owner-uid of a record

  • membership (str) – resource type which defines membership privacy

  • id (str) – the resource ID if membership is set

Returns

actions available to principal

Return type

set

with_filter(query, access='r')

Apply RBAC and privacy to a query

Parameters
  • query (obj) – a resource query in SQLalchemy

  • access (str) – one of lrwcd (list, read, write, create, delete)

Returns

updated SQLalchemy query with filter applied

Return type

obj

TODO restrictions on contact-read by list-id

with_permission(access, query=None, new_uid=None, membership=None, id=None)

Evaluate permission to access an object identified by an open query or new uid. Pass in at least one of the query/uid/eid params

Parameters
  • access (str) – one of lrwcd (list, read, write, create, delete)

  • query (obj) – a resource query by id in SQLalchemy

  • new_uid (str) – user id of a new record

  • membership (str) – resource type which defines membership privacy

  • id (str) – resource ID

Returns

True if access allowed

Return type

bool

class apicrud.AccountSettings(account_id, db_session=None, uid=None)

Access class for account settings, with cache - converts db record to object attributes

Each account is associated with an entry in the Settings model; this class provides access to these key-value pairs as read-only attributes. Because these functions are called frequently, the db entry is loaded into memory for a configurable expiration period (config.REDIS_TTL).

Parameters
  • account_id (str) – ID in database of a user’s account

  • db_session (obj) – a session connected to database

  • uid (str) – User ID

property locale

Returns the language for the uid if specified in the user’s profile

uncache()

Clear the cached settings for account_id

class apicrud.BasicCRUD(resource=None, model=None)

Controller base class

Create/Read/Update/Delete/find controller operations.

This class provides permission-based, paginated access to database models behind your application’s endpoints. Most endpoints need no boilerplate code, and can inherit these functions directly. Some endpoints only need a few lines of code before or after inheriting these functions. You can always write your own custom function for special-case endpoints.

Parameters
  • resource (str) – a resource name (endpoint prefix)

  • model (obj) – the model corresponding to the resource

static create(body, id_prefix='x-', limit_related={})

Controller for POST endpoints. This method assigns a new object ID, sets the _created_ timestamp, evaluates user’s permissions, adds a default category_id if the model has this attribute, and inserts a row to the back-end database.

Parameters
  • body (dict) – resource fields as defined by openapi.yaml schema

  • id_prefix (str) – generated objects will be assigned a random 10- to 16-character ID; you can set a unique prefix if desired

  • limit_related (dict) – limits on number of related records, keyed by relationship name

Returns

first element is a dict with the id, second element is response code (201 on success)

Return type

tuple

db_get(id)

Activate a SQLalchemy query object for the specified ID in the current model

Parameters

id (str) – object ID

Returns

query object

Return type

obj

static delete(ids, force=False)

Controller for DELETE endpoints. This method looks for existing records, evaluates user’s permissions, and updates or removes rows in the back-end database.

Parameters
  • ids (list of str) – record IDs to be flagged for removal

  • force (bool) – flag for removal if false; remove data if true

Returns

first element is a dict with the id, second element is response code (200 on success)

Return type

tuple

static find(**kwargs)

Find records which match query parameters passed from connexion by name, in a dictionary that also includes user and token info

Parameters
  • cursor_next (str) – pagination token to fetch subsequent records

  • filter (dict) – field/value pairs to query (simple queries only, with string or list matching; or * for any)

  • limit (int) – max records to fetch

  • offset (int) – old-style pagination starting offset

  • sort (str) – <field>[:{asc|desc}]

  • status (str) – value is added to filter

Returns

items (list), count(int), cursor_next (str)

Return type

dict

static get(id)

Controller for GET endpoints. This method evaluates privacy settings against the user’s permissions, looks up category, owner and geocode values, and fetches the object from back-end database.

Parameters

id (str) – ID of the desired resource

Returns

first element is a dict with the object or error message, second element is response code (200 on success)

Return type

tuple

static update(id, body, access='u', limit_related={})

Controller for PUT endpoints. This method looks for an existing record, evaluates user’s permissions, and updates the row in the back-end database.

Parameters
  • body (dict) – fields to be updated

  • access (str) – access-level required for RBAC evaluation

  • limit_related (dict) – limits on number of related records, indexed by relationship name

Returns

first element is a dict with the id, second element is response code (200 on success)

Return type

dict

static update_contact(id, body)

This is a special-case function for the contact-update resource

  • validate sms carrier

  • keep person identity in sync with primary contact

Parameters
  • id (str) – resource ID

  • body (dict) – as defined in openapi.yaml

class apicrud.Grants(db_session=None, ttl=None)

Account usage limits

An account’s usage limits are specified here in the grants table; the free-service tier is defined and passed in via load_defaults(). Records in grants table are owned by administrator-level user. If a record matches a user uid, the default grant name=value is overridden.

db_session

existing db session

Type

obj

ttl

how long to cache a grant in memory

Type

int

crud_get(crud_results, id)

Process results from BasicCRUD.get() for grants endpoint. If the id is found in database, perform the standard CRUD get(). Otherwise, look for a hybrid id in form uid:grant and return the cached Grant value. Grant values are serialized as strings even if they are integers (decimal, octal, hex).

Parameters
  • crud_results (tuple) – preliminary response

  • name (str) – name filter, if specified

find(crud_results, **kwargs)

Process results from BasicCRUD.find() for grants endpoint

Parameters
  • crud_results (tuple) – preliminary response

  • name (str) – name filter, if specified

get(name, uid=None)

Get the cached value of a named grant, if it hasn’t expired

Parameters
  • name (str) – name of a grant, as defined in service config

  • uid (str) – user ID

Returns

granted limit or None if undefined

Return type

int or str

load_defaults(defaults)

Load default values from a dict of keyword: value pairs

Parameters

defaults (dict) – new defaults

uncache(uid)

Remove grants from cache, any time a user’s status changes

Parameters

uid (str) – user ID

class apicrud.Metrics(uid=None, db_session=None, func_send=None)

This implementation supports standard system metrics and two types of usage-billing: granted limits per period, or metered usage.

See this article for a description of how redis makes the implementation straightforward: https://www.infoworld.com/article/3230455/how-to-use-redis-for-real-time-metering-applications.html The data store is in-memory, with snapshot/append persistence to disk.

All metrics are defined in the metrics section of service_config.yaml. These follow a de facto standard described in best-practices documentation section of prometheus.io website. At time of implementation, flask prometheus client is not mature/user-friendly enough or suitable for usage-billing so to keep this consistent with service_config.yaml, there is no intent now or in the future to use it.

Thanks to the expiring keys feature of redis, grant limits can be implemented without any periodic cleanup task. Upon first use, a decrementing key is created with expiration set to the configured period. If it counts down to zero before expiration (for example, 50 video uploads per day), the user is denied access to the resource until the key expires. If a user never returns, there will be no redis keys consuming system resources unless and until the user comes back, at which point a new grant period starts. Scaling to tens of millions of users is practical within a small footprint, and this is why user-tracking metrics are stored in redis rather than a traditional database.

uid

ID of a user, for usage tracking

Type

str

db_session

existing db connection

Type

obj

func_send

function name for sending message via celery

Type

obj

check(name)

Check remaining credit against grant-style metric

Params:

name (str): a metric name

Returns

amount of credit remaining, or None

Return type

float

collect(**kwargs)

Prometheus-compatible metrics exporter. Collects current metrics from redis, evaluates metric-type settings in service_config.yaml, and returns a plain-text result in this form:

# TYPE process_resident_memory_bytes gauge process_resident_memory_bytes{instance=”k8s-01.ci.net”} 20576.0 # TYPE process_start_time_seconds gauge process_start_time_seconds{instance=”k8s-01.ci.net”} 1614051156.14 # TYPE api_calls_total counter api_calls_total{resource=”person”} 4

find(**kwargs)

Look up metrics defined by filter

Returns

tuple (results dict, status)

store(name, labels=[], value=None)

Store a metric: redis keys are in form mtr:<name>:<labels> To avoid possible ambiguity and multiple redis keys for the same label set, this function sorts labels before storing.

Params:

name (str): a metric name labels (list): labels, usually in form <label>=<value> value (int or float): value to store

Returns

true in all cases except for grant exceeded

Return type

bool

class apicrud.Mutex(lockname, redis_host=None, maxwait=20, ttl=0, redis_conn=None)

Simple mutex implementation for non-clustered Redis

Parameters
  • lockname (str) – a unique name for the lock

  • redis_host (str) – IP or DNS name of redis service

  • maxwait (int) – seconds to wait for a lock

  • ttl (int) – seconds to hold lock

  • redis_conn (obj) – existing redis connection

acquire()

Acquire a mutex lock

Raises

TimeoutError – if the resource is unavailable

release()

Release a lock

class apicrud.RateLimit

Rate Limiting

call(service='*', uid=None)

Apply the granted rate limit for uid to a given service Two redis keys are used, for even and odd intervals as measured modulo the Unix epoch time. The current interval’s key is incremented if we haven’t reached the limit yet, then we delete that key at first call seen next interval and increment the other key. The keys expire within interval-1 seconds to keep the cache clear after the user stops sending calls.

Parameters
  • service (str) – name of a service

  • uid (str) – a user ID

Returns

true if limit exceeded

Return type

bool

class apicrud.ServiceConfig(file=None, models=None, reset=False, **kwargs)

Service configration - have it your way!

Flask and application configuration global default values are defined in service_config.yaml here in the source directory. Overrides of these values can be specified in a few ways and are evaluated in this order:

  • Environment variables set by parent process

  • Values defined in a file in yaml format

  • Values passed as a keyword arg at first class invocation

For runtime security, the config singleton is stored as an immutable namedtuple: to change values, update settings and restart the container running the service.

An endpoint /config/v1/config provides read-only access to these values except those of type password. Always override those secret values before deploying your service.

Attribute keys specified as env vars are UPPERCASE, and attribute keys stored in the config object are also uppercase. Use lowercase to specify attribute keys in kwargs or the yaml input file.

Parameters
  • file (str) – path of a YAML file defining override values

  • models (obj) – sqlalchemy db models

  • reset (boolean) – reset cached values (for unit tests)

  • **kwargs – key=value pair arguments to override values

Raises

AttributeError if invalid specification

set(key, value)

Set a single value

Parameters
  • key (str) –

  • - new value (value) –

class apicrud.ServiceRegistry(aes_secret=None, public_url=None, reload_params=False, ttl=None)

Service Registry

Services or the UI discover one another through this service registry. Each microservice instance submits its identity and capabilities to this central registry, implemented as expiring redis keys which are updated at a fixed frequency. Encryption provides modest protection against injection attacks.

Parameters
  • aes_secret (str) – an AES secret [default: config.REDIS_AES_SECRET]

  • public_url (str) – URL to serve [default: config.PUBLIC_URL]

  • reload_params (bool) – force param reload, for unit-testing

  • ttl (int) – how long to cache instance’s registration

find(service_name=None)

Finds one or all services

Parameters

service_name (str) – a service, or None for all

Returns

dict - instances (list of registered services)

url_map (public url for each top-level resource)

get()

return service registration for local instance

Returns: dict(name, id, info)

Key info is dict(endpoints, ipv4, port, public_url, created)

register(resource_endpoints, service_name=None, instance_id='build-13644181-project-614839-apicrud-media', tcp_port=None)

register an instance serving a list of endpoints

Parameters
  • resource_endpoints (list of str) – controller endpoints served

  • service_name (str) – microservice name

  • instance_id (str) – unique ID of instance

  • tcp_port (int) – port number of service

static update()

background function to update registration at the defined interval from local memory cache, until the instance terminates.

class apicrud.SessionAuth(roles_from=None)

Session Authorization

Functions for login, password and role authorization

Parameters

roles_from (obj) – model for which to look up authorizations

account_add(username, uid)

Add an account with the given username

Parameters
  • username (str) – new / unique username

  • uid (str) – existing user

account_login(username, password, method='local', otp=None)

Log in using local or OAuth2 credentials

Parameters
  • username (str) – account name or email

  • password (str) – credential

  • method (str) – local, or google / facebook / twitter etc

  • otp (str) – one-time or backup password

Returns

Fields include jwt_token (contains uid / account ID), ID of entry in settings database, and a sub-dictionary with mapping of endpoints registered to microservices

Return type

dict

auth_params()

Get authorization info

get_roles(uid, member_model=None, resource=None, id=None)

Get roles that match uid / id for a resource Each is in the form <resource>-<id>-<privacy level>

Parameters
  • uid (str) – User ID

  • member_model (str) – resource-name of DB model that defines membership in resource

  • resource (str) – the resource that defines privacy (e.g. list)

  • id (str) – ID of the resource (omit if all are desired)

Returns

authorized roles

Return type

list of str

methods()

Return list of available auth methods

totp_bypass(uid)

Check for bypass cookie

Parameters

uid (str) – User ID

Returns

valid bypass found

Return type

bool

update_auth(id, member_model=None, resource=None, force=False)

Check current access, update if recently changed

Parameters
  • id (str) – resource id of parent resource

  • resource (str) – parent resource for which membership should be checked

  • force (bool) – perform update regardless of logged-in permissions

class apicrud.SessionManager(ttl=None, redis_conn=None)

Session Manager - for active user sessions

Each login session is stored as an encrypted JSON dict in redis, indexed by sub:token

Parameters
  • ttl (int) – seconds until a session expires

  • redis_conn (obj) – connection to redis service

create(uid, roles, key_id=None, **kwargs)

Create a session, which is an encrypted JSON object with the values defined in https://tools.ietf.org/html/rfc7519 for JWT claim names:

  • exp - expiration time, as integer Unix epoch time

  • iss - a constant JWT_ISSUER

  • jti - JWT ID, the randomly-generated token

  • sub - the uid of a user

We add these:

  • auth - authorized roles

  • any other key=value pairs the caller passes as kwargs

The session automatically expires based on object’s ttl. Part of the jti token is used in redis key, to allow a user to log into multiple sessions. The rest of the token is encrypted, to secure it from replay attack in the event redis traffic is compromised.

Parameters
  • uid – User ID

  • roles – Authorized roles

  • key_id – session key ID for redis (defaults to uid)

  • nonce – a unique identifier for the token (random if not specified)

  • ttl – duration of session (defaulted from class init)

Returns

Keys include auth (authorized roles), exp / iss / jti / sub (as above), along with parameters passed into this function

Return type

dict

delete(uid, token, key_id=None)

Cancel a session

Parameters
  • uid – User ID

  • token (str) – The token value passed from create as ‘jti’

  • key_id (str) – session key ID for redis

get(uid, token, arg=None, key_id=None)

Get one or all key-value pairs stored by session create

Parameters
  • uid (str) – User ID

  • token (str) – The token value passed from create as ‘jti’

  • arg (str) – key of desired value (None to fetch all)

  • key_id (str) – session key ID for redis (defaults to uid)

Returns

single value or dictionary of all session keys

Return type

dict or str

update(uid, token, arg, value, key_id=None)

Update a specified session key

Parameters
  • uid – User ID

  • token (str) – The token value passed from create as ‘jti’

  • arg (str) – key to update

  • value (str) – new value for key

  • key_id (str) – session key ID for redis (defaults to uid)

basic_crud

basic_crud.py

media