Commit e9c455ff authored by 20after4's avatar 20after4
Browse files

Code cleanup & improved documentation in README and docs/*.md

parent 6814d392
......@@ -41,7 +41,10 @@ setup.py will install a command line tool called `dddcli`
To install for development use:
```bash
python3 setup.py develop
pip3 install virtualenv poetry
virtualenv --python=python3 .venv
source .venv/bin/activate
poetry install
```
### dddcli
......@@ -117,6 +120,10 @@ Datasette has been extended with some plugins to add custom functionality.
`src/datacube-dashboards`. Do the usual `git submodule update --init` to get that source code.
* There are custom views and routes added in ddd_datasette.py that map urls like /-/ddd/$page/ to files in `www/templates/view/`.
# Dashboards
The data³ Dashboards web application is documented in [./docs/DefiningDashboards.md](docs/DefiningDashboards.md).
# Example code:
## Conduit API client:
......
# Data³ Dashboards
Dashboards in Data³ are built in html by web-components implementing a few custom html elemeents.
The dashboard UI consists of a query form across the top of the page which is the primary navigation element. Editing any of the query fields will change state which is kept in the url and in the javascript code that implements the reactive application.
## Application Structure
A good place to start understanding the application is with an overview of the key components and their roles:
The dashboard page is built from the view template in [dashboard.html](../www/templates/views/dashboard.html)
* Most of the application is implemented in typescript which is compiled into javascript and then rolled up into a single app.js file called `static/app.js`.
* The application is loaded by `require.js` which is called from a script tag in dashboard.html.
* Once app.js is loaded the custom elements are bound to their javascript implementations and the components initialize in the order which they are referenced in the `initApp()` function in [DashboardApp.ts](../www/static/DashboardApp.ts)
* DashboardApp.ts - the "main" entrypoint for the application. Deals with initializing the application and facilitates coordination between the other components.
* The navigation ui is implemented by classes in filter-input.ts, specifically `AutocompleteFilter` and `DaterangeFilter`.
* The dashboard charts are implemented by passing a vega lite spec to vega-embed which is wrapped by the web component class named `VegaChart`. See [vega-tonic.ts](../www/static/vega-tonic.ts).
* The data for the charts is provided by DataSource instances.
** Each DataSource is defined with a sql query template.
** The query can contain :placeholders which are filled in at runtime with the values from corresponding url query parameters.
** When the url state changes, any data DataSource which references the changed variable will be notified to update.
** Each affected DataSource will then fetch fresh json from the back-end database. When finished fetching data, then dependent charts are notified to update and provided with the new data.
---
I've attempted to describe the application state and control flow with the following diagram:
### State Flowchart
```mermaid
flowchart TB
classDef dashed stroke:#ccc,stroke-width:2px,color:#fff,stroke-dasharray: 5 5;
User-->|interaction| InputFilter
InputFilter("InputFilter (Query filter user interface)")
Query{{Query state}}
subgraph DataSources
DataSource{{<data-source>}}
db[(DataSette back-end)]
SQL{{Parameterized SQL}}
SQL-- execute query -->db;
db --> |result json| DataSource;
DataSource-- :parameters -->SQL;
class SQL dashed
end
InputFilter-- query filter state change-->Query;
URL(Browser URL)-- popState / setState -->Query;
Query-- history.pushState -->URL;
Query-- :parameters -->DataSource
subgraph Charts[Charts]
DataSource--> |query results|a;
a{{<vega-chart>}} --> VegaChart
VegaChart -->|vega-lite spec + data| VegaEmbed(Vega-Embed renderer);
subgraph VegaChart[VegaChart instances...]
spec{{vega-lite spec www/views/charts/*.yaml}}
class layout,databind dashed
end
class VegaChart dashed
end
```
-----
## Adding new charts to the dashboard:
To add a new chart it's probably easiest to start from an existing example. A good starting point would be [leadtime.yaml](www/templates/views/charts/leadtime.yaml). So start by making a copy of leadtime.yaml under a different name.
The yaml structure controls the positioning of the chart as well as the vega spec which maps query columns to axes on the chart.
### Example:
-----
```yaml
# The first part of the yaml defines the name of the chart,
# the database ("metrics.db") and the query name that will be
# used to get the data for the chart.
title: Lead & Cycle Time Histogram
db: metrics
tab: charts
order: 4 # the order of the chart, relative to other
# charts on the page.
query: cycletime # the name of the query, this will read the
# query's sql definition from a file called
# cycletime.sql
type: vega # this tells dashboard to use the vega-embed
# library to render the chart.
# Everything within the display section defines a
# vega-lite specification. vega-lite is normally specified in
# json format and to satisfy the vega compiler we produce json. # This yaml is directly converted to json by
# parsing with the python yaml parser and then encoding the
# resulting structure using the python json encoder.
display:
# example vega-lite view specification formatted as yaml:
width: 400
height: 300
mark:
type: bar
tooltip: true
encoding:
x:
field: duration
type: ordinal
bin:
maxbins: 20
title: Cycle time (days, binned)
y:
aggregate: count
title: Count of tasks
color:
field: duration
scale:
scheme: browns
legend: null
```
-----
To learn more about the vega view specification language you can read about it in the [vega-lite documentation](https://vega.github.io/vega-lite/docs/spec.html) or browse some [examples](https://vega.github.io/vega-lite/examples/).
\ No newline at end of file
......@@ -2,56 +2,61 @@
This type stub file was generated by pyright.
"""
from typing import Optional
from datasette.app import Datasette
ureg = ...
class DatasetteError(Exception):
def __init__(self, message, title=..., error_dict=..., status=..., template=..., message_is_html=...) -> None:
...
class BaseView:
ds = ...
def __init__(self, datasette) -> None:
ds:Optional[Datasette] = None
def __init__(self, datasette:Datasette) -> None:
...
async def head(self, *args, **kwargs):
...
async def check_permission(self, request, action, resource=...): # -> None:
...
async def check_permissions(self, request, permissions): # -> None:
"""permissions is a list of (action, resource) tuples or 'action' strings"""
...
def database_color(self, database): # -> Literal['ff0000']:
...
async def options(self, request, *args, **kwargs): # -> Response:
...
async def post(self, request, *args, **kwargs): # -> Response:
...
async def put(self, request, *args, **kwargs): # -> Response:
...
async def patch(self, request, *args, **kwargs): # -> Response:
...
async def delete(self, request, *args, **kwargs): # -> Response:
...
async def dispatch_request(self, request, *args, **kwargs): # -> Any:
...
async def render(self, templates, request, context=...): # -> Response:
...
@classmethod
def as_view(cls, *class_args, **class_kwargs): # -> (request: Unknown, send: Unknown) -> Coroutine[Any, Any, Any]:
...
class DataView(BaseView):
......@@ -59,25 +64,25 @@ class DataView(BaseView):
re_named_parameter = ...
async def options(self, request, *args, **kwargs): # -> Response:
...
def redirect(self, request, path, forward_querystring=..., remove_args=...): # -> Response:
...
async def data(self, request, database, hash, **kwargs):
...
async def resolve_db_name(self, request, db_name, **kwargs): # -> tuple[Unknown, Unknown | Literal['000'], Unknown, Unknown] | tuple[Unknown, Unknown | Literal['000'], Unknown, None]:
...
def get_templates(self, database, table=...): # -> None:
...
async def get(self, request, db_name, **kwargs): # -> Response | AsgiStream:
...
async def as_csv(self, request, database, hash, **kwargs): # -> Response | AsgiStream:
...
async def get_format(self, request, database, args): # -> tuple[Unknown | str | None, Unknown]:
"""Determine the format of the response from the request, from URL
parameters or from a file extension.
......@@ -85,12 +90,12 @@ class DataView(BaseView):
`args` is a dict of the path components parsed from the URL by the router.
"""
...
async def view_get(self, request, database, hash, correct_hash_provided, **kwargs): # -> Response | AsgiStream:
...
def set_response_headers(self, response, ttl):
...
......@@ -192,17 +192,6 @@ def extra_template_vars(template:str, database:str, table:str, columns:str, view
}
# def magic_phid(key, request):
# return 'mmodell'
# @hookimpl
# def register_magic_parameters(datasette):
# return [
# ("phid", magic_phid),
# ]
@hookimpl
def canned_queries(datasette: Datasette, database: str) -> Mapping[str, str]:
# load "canned queries" from the filesystem under
......
......@@ -22,9 +22,7 @@ function initApp() {
Tonic.add(DashboardApp);
initDataSets();
const app = <DashboardApp> <unknown>document.getElementsByTagName('dashboard-app')[0];
console.log(TaskDialog);
console.log('---------------- init ----------------')
//console.log('---------------- init ----------------')
}
......@@ -36,8 +34,6 @@ class DashboardApp extends DependableComponent {
constructor() {
super();
this.query = Query.init();
this.addEventListener('change', this.change);
const form = this.querySelector('form');
......@@ -54,7 +50,6 @@ class DashboardApp extends DependableComponent {
this.setState(this.query);
const loaded = (event?) => {
console.log("Content loaded");
for (const chart of document.querySelectorAll('vega-chart')) {
(chart as unknown as VegaChart).loadcharts();
}
......@@ -82,7 +77,7 @@ class DashboardApp extends DependableComponent {
}
this.query.set(e.target.id, e.target.value)
console.log('changed', e.target.id, e.target.value);
this.debug('changed', e.target.id, e.target.value);
this.update_state_listeners();
}
......@@ -98,16 +93,14 @@ class DashboardApp extends DependableComponent {
}
setState(state=null) {
console.log('setState');
//this.update_state_listeners();
for (const ele of this.querySelectorAll('.filter') as any as InputFilter[]) {
console.log('setState',ele,this.query);
this.debug('setState',ele,this.query);
ele.setState(this.query);
}
}
submit(e?){
//const e = arguments[0];
if (e) {
e.preventDefault();
e.stopPropagation();
......@@ -141,7 +134,7 @@ class DashboardApp extends DependableComponent {
}
async loadContent(url:URL) {
console.log('loadContent', url);
this.debug('loadContent', url);
// for (const ele of document.querySelectorAll('data-set')) {
// const ds = ele as unknown as DataSet;
......@@ -150,7 +143,7 @@ class DashboardApp extends DependableComponent {
var reportUrl = new URL(url);
reportUrl.pathname = url.pathname.replace('dashboard/project-metric', 'cycletime/')
console.log('reportUrl', reportUrl.href);
this.debug('reportUrl', reportUrl.href);
const response = await fetch(reportUrl.href);
if (response.status === 200) {
const tmpl = document.createElement('template');
......
This diff is collapsed.
......@@ -122,7 +122,7 @@ class BaseDataSet extends DependableComponent {
this.query = Query.init();
}
stateChanged(key, values) {
console.log('stateChanged on', this, key, values);
this.debug('stateChanged on', this, key, values);
this.state.replacer = param_replacer(this.query);
this.reRender();
......@@ -133,7 +133,7 @@ class BaseDataSet extends DependableComponent {
(ele as DatasetConsumer).datasetChanged(this);
}
} catch(err) {
console.error(err, ele);
this.error(err, ele);
}
}
}
......@@ -173,7 +173,7 @@ class DataSet extends BaseDataSet {
url.searchParams.set('sql', sql.toString());
const query_params = this.query.state;
console.log('query_params', query_params, sql);
this.debug('query_params', query_params, sql);
for (const prop in query_params) {
var val = query_params[prop];
if (!val){
......@@ -195,6 +195,7 @@ class DataSet extends BaseDataSet {
}
return url.href;
}
async fetch(cache=true){
const url = this.url;
......@@ -228,12 +229,28 @@ class DataSet extends BaseDataSet {
class StaticDataSet extends DataSet {
async fetch() {
const url = this.props.url;
const response = await fetch(url);
async fetch(ids) {
const url = new URL(this.props.url, window.location.protocol+'//'+window.location.host);
if (this.props.sql) {
var sql:string = this.props.sql;
if (ids) {
var placeholders = [];
var i = 0;
for (var i = 1; i <= ids.length; i++) {
placeholders.push(":id"+i);
url.searchParams.set('id'+i, ids[i-1]);
}
sql = sql.replace('?*', placeholders.join(','));
}
url.searchParams.set('sql', sql);
}
const response = await fetch(url.href);
const fetched = await response.json();
this.state.data = new DatasetCursor(this, fetched, url);
this.state.data = new DatasetCursor(this, fetched, url.href);
return this.state.data;
}
render (p1, p2, p3) {
......@@ -241,6 +258,82 @@ class StaticDataSet extends DataSet {
}
}
interface DataMap {
[key: string]:any
}
function makeTimeoutPromise(data={}, time_ms=1) {
return new Promise(async (resolve, reject) => {
setTimeout(()=>{
resolve(data);
}, time_ms)
});
}
/**
* AsyncComponentFetcher -
* handles loading a bunch of objects from the back end
* database by batching individual calls to load() and bundling them together into
* one call to the backend where the rows are loaded in a single query of the form:
* SELECT * FROM Table WHERE pk IN (id1, id2, ..., idn)
*
* The async load(id) function blocks up to 10ms to allow a bunch of async calls to collect
* the ids that are needed and then load will batch the calls into one fetch, finally the
* results of the fetch are unbundled and returned by the individual promises.
*/
class AsyncComponentFetcher {
ds:StaticDataSet;
cls:typeof DependableComponent;
data:DataMap;
pending:Promise<any>;
batching:Promise<any>;
pk:string;
requestedIds:string[] = [];
constructor(ds:StaticDataSet, cls:typeof DependableComponent, pk:string) {
this.ds = ds;
this.cls = cls;
this.data = {};
this.pk = pk;
}
async load(id){
if (this.data[id]) {
return this.data[id];
}
if (this.requestedIds.indexOf(id) == -1 ) {
this.requestedIds.push(id);
}
if (!this.batching) {
this.batching = makeTimeoutPromise({}, 10);
await this.batching;
const ids = this.requestedIds;
if (ids.length) {
this.requestedIds = [];
const pending = new Promise(async (resolve, reject) => {
const data = await this.ds.fetch(ids);
for (const obj of data.rows) {
this.data[obj[this.pk]] = new this.cls(obj);
}
resolve(data);
this.pending = null;
});
this.pending = pending;
await pending;
}
this.batching = null;
} else {
await this.batching;
}
if (this.pending) {
await this.pending;
}
return this.data[id];
}
}
interface DataResponse {
rows: any[];
columns: string[];
......@@ -330,8 +423,7 @@ async function fetchData(dataset_id:string, cache=true, wait=true) {
const ds = <DataSet> <unknown>document.getElementById(dataset_id);
if (!ds) {
console.log('dataset not found: ', dataset_id, ds);
return;
throw new Error(`dataset not found: ${dataset_id}`)
}
const promise = ds.fetch();
if (wait) {
......@@ -342,4 +434,4 @@ async function fetchData(dataset_id:string, cache=true, wait=true) {
}
export {DataSource, BaseDataSet, DataSet, StaticDataSet, initDataSets, fetchData}
\ No newline at end of file
export {DataSource, BaseDataSet, DataSet, StaticDataSet, initDataSets, fetchData, AsyncComponentFetcher}
\ No newline at end of file
......@@ -77,7 +77,7 @@ class Query {
const state = this._state;
for (const key in newState) {
const val = newState[key];
console.log('state['+key+']='+val);
//console.log('state['+key+']='+val);
if (state[key] && state[key] !== val || !state[key]) {
this.set(key, val);
}
......@@ -164,7 +164,7 @@ class DependableComponent extends Tonic {
hasResolved = false;
static _waitingFor:WaitingForId = { };
static debug_logging = true;
static debug_logging = false;
static logging = true;
debug(...args) {
if (DependableComponent.debug_logging){
......@@ -179,8 +179,11 @@ class DependableComponent extends Tonic {
console.log(...args);
}
}
constructor() {
constructor(data=null) {
super();
if (data) {
this.state.data = data;
}
}
get base_url() {
return window['BASE_URL'] || DependableComponent._base_url;
......
......@@ -567,6 +567,31 @@ class NavTabs extends InputFilter {
}
}
keyup(e) {
console.log(e);
var next;
if (e.key == 'ArrowRight') {
next = e.target.parentElement.nextElementSibling;
} else if (e.key == 'ArrowLeft') {
next = e.target.parentElement.previousElementSibling;
} else if (e.key == ' ') {
e.preventDefault();
e.stopPropagation();
e.target.click();
}
if (!next) {
return;
}
console.log(next);
if (next && next.getAttribute('role') == 'tab') {
const anchor = next.querySelector('a');
anchor.click();
anchor.focus();
e.preventDefault();
return;
}
}
click(e){
if (!e.target.matches('.nav-link')) return;
e.preventDefault();
......@@ -585,23 +610,21 @@ class NavTabs extends InputFilter {
const tabs = this.tabs.map(tab=>{
const selected = tab.selected ? 'active':'';
const label = tab.getAttribute('label');
const aria = tab.selected ? 'aria-current="page"' : '';
const aria_selected = tab.selected ? "true" : "false";
return this.html`
<li class='nav-item' ${aria} panel='${tab.id}'>
<a class='nav-link ${selected}' href='#${tab.id}'>${label}</a>
<li class='nav-item' role='tab' aria-selected='${aria_selected}' aria-controls='${tab.id}' panel='${tab.id}'>
<a class='nav-link ${selected}' tabindex='0' href='#${tab.id}'>${label}</a>
</li>`;
});
return tabs;
}
render() {
console.log('this.tabs', this.tabs);
return this.html`
<ul class="nav nav-tabs">
<ul class="nav nav-tabs" role="tablist">
${this.renderTabs()}
</ul>
<div class='tab-panels'>
<div class="tab-panels">