Commit 54eb5781 authored by 20after4's avatar 20after4 Committed by Mhurd
Browse files

A whole bunch more changes for jinja views / dashboards

parent 623f51e8
__pycache__
.mypy_cache
.pytest_cache
*.bak
*.old
*.csv
*.pyc
.ipynb_checkpoints
......@@ -9,8 +11,10 @@ __pycache__
*.egg-info
*.db
/.eggs
/build
/dist
/.venv
/test/*.json
/ddd.code-workspace
/ignore/*
/node_modules/
[submodule "src/datasette-dashboards"]
path = src/datasette-dashboards
url = https://gitlab.wikimedia.org/20after4/datasette-dashboards.git
[submodule "src/datasette"]
path = src/datasette
url = https://gitlab.wikimedia.org/twentyafterfour/datasette.git
# D³
# Data³
`ddd` or `d³` is a toolkit for accessing APIs and processing data from disperate
systems.
Data³ is a toolkit and general framework for visualizing just about any data. Wikimedia's engineering productivity team have begun assembling a toolkit to help us organize, analyze and visualize data collected from our development, deployment, testing and project planning processes. There is a need for better tooling and data collection in order to have reliable and accessible data to inform data-driven decision-making. This is important because we need to measure the impact of changes to our deployment processes and team practices so that we can know whether a change to our process is beneficial and quantify the impacts of the changes we make.
The first applications for the Data³ tools are focused on exploring software development and deployment data, as well as workflow metrics exported from Wikimedia's phabricator instance.
The core of the toolkit consists of the following:
* Datasette.io provides a front-end for browsing and querying one or more SQLite databases.
* A customized version of datasette-dashboards is included for visualizing the output of queries in Vega/Vega-Lite charts and it can render jinja templates for custom reports or interactive displays.
* A comprehensive python library and command line interface for querying and processing Phabricator task data exported via conduit api requests.
* Several custom dashboards for datasette which provide visualization of metrics related to Phabricator tasks and workflows.
* A custom dashboard to explore data and statistics about production MediaWiki deployments.
## Status
......@@ -37,10 +46,9 @@ You can use the following sub-commands by running `dddcli sub-command [args]` to
### Phabricator metrics: `dddcli metrics`
This tool is used to extract data from phabricator and organize it in a structure that will facilitate further analysis.
The analysis of task activities can provide some insight into workflows.
The output if this tool will be used as the data source for charts to visualize certain agile project planning metrics.
* This tool is used to extract data from phabricator and organize it in a structure that will facilitate further analysis.
* The analysis of task activities can provide some insight into workflows.
* The output if this tool will be used as the data source for charts to visualize certain agile project planning metrics.
#### cache-columns
The first thing to do is cache the columns for the project you're interested in.
......@@ -56,10 +64,10 @@ Then you can fetch the actual metrics and map them into local sqlite tables with
```bash
dddcli metrics map --project=PHID-PROJ-uier7rukzszoewbhj7ja
dddcli metrics map --project=#release-engineering-team
```
Note that `--project` accepts either a `PHID` or a project `#hashtag`, so you can try `dddcli metrics map --project=#releng`, for example.
Note that `--project` accepts either a `PHID` or a project `#hashtag`
To get cli usage help, try
......@@ -77,17 +85,33 @@ This runs the mapper with data from a file, treating that as a mock api call res
If you omit the --mock argument then it will request a rather large amount of data from the phabricator API which takes an extra 20+ seconds to fetch.
### datasette
### Datasette
The main user interface for the Data³ tool is provided by Datasette.
To run datasette, from the ddd checkout:
Datasette is installed as a dependency of this repo by running `poetry install` from the repository root.
Once dependencies are installed, you can run datasette from the ddd checkout like this:
```bash
export DATASETTE_PORT=8001
datasette --reload --metadata www/metadata.yaml -h 0.0.0.0 -p $DATASETTE_PORT www
export DATASETTE_HOST=localhost # or use 0.0.0.0 to listen on a public interface
export DATASETTE_DIR=./www #this should point to the www directory included in this repo.
datasette --reload --metadata www/metadata.yaml -h #DATASETTE_HOST -p $DATASETTE_PORT $DATASETTE_DIR
```
Sample systemd units are in `etc/systemd/*` including a file watcher to restart datasette
when the data changes.
For deployment on a server, there are sample systemd units in `etc/systemd/*` including a file watcher to
restart datasette when the data changes. Approximately the same behavior is achieved by the --reload argument to the
datasette command given here and that is adequate for development and testing locally.
### Datasette Plugins
Datasette has been extended with some plugins to add custom functionality.
* See `www/plugins` for Data³ customizations.
* There is also a customized version of datasette-dashboards which is included via a submodule at
`src/datacube-dashboards`. Do the usual `git submodule update --init` to get that source code.
* There are custom views and routes added in ddd_datasette.py that map urls like /-/ddd/$page/ to files in `www/templates/view/`.
# Example code:
......
datasette --reload --metadata www/metadata.yaml -h 0.0.0.0 -p 8001 www
\ No newline at end of file
This diff is collapsed.
{
"dependencies": {
"xhr": "^2.6.0"
},
"devDependencies": {
"@babel/core": "^7.15.8",
"@babel/preset-env": "^7.15.8",
"@babel/preset-react": "^7.14.5",
"babel-preset-es2015": "^6.24.1",
"babel-preset-react": "^6.24.1",
"babel-preset-stage-0": "^6.24.1",
"babelify": "^10.0.0",
"browserify": "^17.0.0",
"es2015": "0.0.0"
},
"babel": {
"presets": [
"@babel/preset-env",
"@babel/preset-react"
]
}
}
This diff is collapsed.
......@@ -12,14 +12,14 @@ packages = [
[tool.poetry.dependencies]
python = '^3.7'
typer = {extras = ["all"], version = "^0.3.2"}
datasette = "^0.58.1"
datasette = { path = "src/datasette", develop = true }
datasette-dashboards = { path = "src/datasette-dashboards", develop = true }
click = "<7.2"
semver = "^2.13.0"
requests = "^2.26.0"
sqlite-utils = "^3.17"
rich = "^10.11.0"
regex = "2021.10.8"
[tool.poetry.dev-dependencies]
black = "^21.9b0"
......@@ -45,4 +45,4 @@ dddcli = "ddd.main:cli"
reportMissingTypeStubs = true
[tool.mypy]
implicit_reexport = true
\ No newline at end of file
implicit_reexport = true
Subproject commit dff9795fa319d9cbf29b3ff97a22209d90cd7fff
Subproject commit bf224aa117be3f9c882ce4de27b09f8fa3dccb8c
Subproject commit 944e256d653b723e171e3a2722a5478bbc41e0ed
This diff is collapsed.
This diff is collapsed.
This diff is collapsed.
......@@ -131,8 +131,15 @@ class Conduit(object):
self, queryKey="all", constraints: MutableMapping = {}
) -> Cursor:
"""Find projects"""
query = {}
if not constraints or not len(constraints) and queryKey is not None:
query["queryKey"] = queryKey
if len(constraints):
query["constraints"] = constraints
return self.request(
"project.search", {"queryKey": queryKey, "constraints": constraints}
"project.search", query
)
def maniphest_search(self, constraints: MutableMapping = {}) -> Cursor:
......
This diff is collapsed.
{
"compilerOptions": {
"module": "es2015",
"target": "es2015",
"jsx": "preserve",
"strictFunctionTypes": true,
"sourceMap": true,
"baseUrl": "./www",
"paths": {
"static/*": ["static/*"],
},
},
}
......@@ -3,7 +3,7 @@ This type stub file was generated by pyright.
"""
from datasette.version import __version__ as __version__
from datasette.version __version_info__ as __version_info__
from datasette.version import __version_info__ as __version_info__
from .hookspecs import hookimpl as hookimpl
from .hookspecs import hookspec as hookspec
......@@ -2,7 +2,7 @@
This type stub file was generated by pyright.
"""
from datasette.database import Database
from datasette.database import Database, Results
from .utils.asgi import NotFound
app_root = ...
......@@ -84,10 +84,10 @@ class Datasette:
"""Check permissions using the permissions_allowed plugin hook"""
...
async def execute(self, db_name, sql, params=..., truncate=..., custom_time_limit=..., page_size=..., log_sql_errors=...):
async def execute(self, db_name:str, sql:str, params=..., truncate=..., custom_time_limit=..., page_size=..., log_sql_errors=...) -> Results:
...
async def expand_foreign_keys(self, database, table, column, values): # -> dict[tuple[Unknown, Unknown], str]:
async def expand_foreign_keys(self, database:str, table:str, column:str, values): # -> dict[tuple[str, Any], str]:
"""Returns dict mapping (column, value) -> label"""
...
......
This diff is collapsed.
This diff is collapsed.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment