ddd merge requestshttps://gitlab.wikimedia.org/repos/releng/ddd/-/merge_requests2021-12-23T23:24:59Zhttps://gitlab.wikimedia.org/repos/releng/ddd/-/merge_requests/9Implement dashboards as web components2021-12-23T23:24:59Z20after4Implement dashboards as web components/-/ddd/dashboard/* is now a collection of web components in typescript.
`www/static/DashboardApp.ts` is the entry point and the main controller of the dashboard UI.
The other major parts are:
- the chart widget is in `vega-tonic.ts`
- .../-/ddd/dashboard/* is now a collection of web components in typescript.
`www/static/DashboardApp.ts` is the entry point and the main controller of the dashboard UI.
The other major parts are:
- the chart widget is in `vega-tonic.ts`
- the search form / filter interface is in `filter-input.ts`
Other significant changes:
- Move charts to separate yaml files and queries to separate sql files
-- Instead of everything in one big yaml file
--- move the definition of each chart into `www/templates/views/charts/*.yaml`
--- "canned queries" are now in `www/sql/*.sql`
--- Dashboards and tabs are still defined in `www/metadata.yaml`
Plus some other smaller fixes:
- Make phid links work with base_url in datasette
- fixup ci & deployment issues in `.gitlab-ci.yaml`
- Fix the metrics mapper cli tool to properly handle cursors.
Now it can automatically resume importing at the point where
the previous run left off.https://gitlab.wikimedia.org/repos/releng/ddd/-/merge_requests/6A whole bunch more changes for jinja views / dashboards2021-10-28T01:07:58Z20after4A whole bunch more changes for jinja views / dashboardsThe most important change included here is the view system
in ddd_datasette.py there is a request url router that
delegates to python modules in www/templates/views/*.py and then
passes the variables from the model to a template with a m...The most important change included here is the view system
in ddd_datasette.py there is a request url router that
delegates to python modules in www/templates/views/*.py and then
passes the variables from the model to a template with a matching
filename (except the extension is html)
So the url /-/ddd/$page loads the python module from
`www/templates/views/$page.py`
which returns some variables for the view.
Then the variables are passed to a jinja template named
`www/templates/views/$page.html`
Also, canned queries are now loaded from sql files in www/sql/$db_name/$query_name
Finally, the function `sql()` can be called from templates and it can reference canned queries
(See extra_template_vars in ddd_datasette.py)https://gitlab.wikimedia.org/repos/releng/ddd/-/merge_requests/5Task dashboard2021-10-20T04:43:40ZMhurdTask dashboardBasic task dashboard using Jinja templates.Basic task dashboard using Jinja templates.https://gitlab.wikimedia.org/repos/releng/ddd/-/merge_requests/4Jinja dashboards2021-10-19T01:52:25Z20after4Jinja dashboardsThis supports dashboard widgets as jinja templates in `www/templates`.
The corresponding query in `metadata.yaml` can be an arbitrary sql string
or it can be the name of a "canned query" and either way the query runs with
your dashboard ...This supports dashboard widgets as jinja templates in `www/templates`.
The corresponding query in `metadata.yaml` can be an arbitrary sql string
or it can be the name of a "canned query" and either way the query runs with
your dashboard filters applied and finally the query results are available from
within the template as a list of dicts named `{{data}}`.
See example in `www/templates/subprojects.html` which renders a list of links, each
pointing to the current dashboard but with the project filter altered to the `PHID`
of the selected sub-project.https://gitlab.wikimedia.org/repos/releng/ddd/-/merge_requests/2Refactored metric mapper + datasette2021-10-05T21:18:01Z20after4Refactored metric mapper + datasetteChange-Id: Iaee1c57c946367ebbb01d6c02d1391c18461a194Change-Id: Iaee1c57c946367ebbb01d6c02d1391c18461a194https://gitlab.wikimedia.org/repos/releng/ddd/-/merge_requests/1Build Workboard Metrics with boardmetrics.py2021-08-18T15:01:39Z20after4Build Workboard Metrics with boardmetrics.pyThe output is suitable for generating a timeline with data points for each time a task enters or leaves a column.
If you run the following command, a metrics.db will be created
```bash
./ddd/boardmetrics.py --project=PHID-PROJ-fmcvjrkf...The output is suitable for generating a timeline with data points for each time a task enters or leaves a column.
If you run the following command, a metrics.db will be created
```bash
./ddd/boardmetrics.py --project=PHID-PROJ-fmcvjrkfvvzz3gxavs3a --mock=test/train.transactions.json --dump=json > metrics.json
```
The schema for the sqlite database is as follows:
| schema |
|--------|
| **events** - individual datapoint for each task event (e.g. workboard move, status change |
| `CREATE TABLE events(ts, task, project phid, user phid, event, old, new)` |
| `CREATE UNIQUE INDEX events_pk on events(ts, task, event)` |
| **column_metrics** - metrics summarized by workboard columns |
| `CREATE TABLE column_metrics (trnsid, ts, project phid, column phid, task, type, value)` |
| `CREATE UNIQUE INDEX trnsid on column_metrics(ts, column, task, value)` |
| `CREATE INDEX ts_column_value on column_metrics(column, task, ts, value)` |
| **task_metrics** - metrics summarized by task |
| `CREATE TABLE task_metrics(task, metric phid, next_metric phid, ts, ts2, duration)` |
| `CREATE UNIQUE INDEX task_metric ON task_metrics(task, metric)` |
| **Columns** - duplicates information in phobjects table, will probably remove this in the future |
| `CREATE TABLE columns(project,phid,name,status,proxyPHID,dateCreated,dateModified)` |
| **phobjects** - cache of every phid to resolve object names and status of each referenced phid |
|`CREATE TABLE phobjects (phid PRIMARY KEY,name TEXT,dateCreated timestamp,dateModified timestamp,status TEXT,type TEXT,data TEXT)`|
| `CREATE INDEX phid_name ON phobjects (phid, name)` |
| `CREATE UNIQUE INDEX pk_phobjects ON phobjects (phid)` |
| `CREATE INDEX type_status ON phobjects (type, status) ` |