🚧 This instance is under construction; expect occasional downtime. Runners available in /repos. Questions? Ask in #wikimedia-gitlab on libera.chat, or under GitLab on Phabricator.

Commit 7b741c67 authored by 20after4's avatar 20after4 🎵
Browse files

Jinja dashboards

parent 27f5a95e
__pycache__
.mypy_cache
.pytest_cache
*.csv
*.pyc
.ipynb_checkpoints
.spyproject
.vscode
cache.db
*.egg-info
*.db
/.eggs
/dist
*.db
/.venv/*
/.venv
/test/*.json
/ddd.code-workspace
/ignore/*
......@@ -6,7 +6,7 @@
## Status
This tool an supporting libraries are in early stages of experimentation and
development. The APIs are not yet stable and the featureset is not yet decoded
development. The APIs are not yet stable and the featureset is not yet decided
let alone completely implemented. Stay tuned or get involved.
## Currently supported data sources:
......@@ -21,7 +21,7 @@ let alone completely implemented. Stay tuned or get involved.
# Usage
## cli
## Installation
setup.py will install a command line tool called `dddcli`
......@@ -31,17 +31,18 @@ To install for development use:
python3 setup.py develop
```
### dddcli
You can use the following sub-commands with `dddcli command [args]` to access various functionality.
You can use the following sub-commands by running `dddcli sub-command [args]` to access various functionality.
### phabricator metrics
### Phabricator metrics: `dddcli metrics`
This tool is used to extract data from phabricator and organize it in a structure that will facilitate further analysis.
The analysis of task activities can provide some insight into workflows.
The output if this tool will be used as the data source for charts to visualize certain agile project planning metrics.
Example usage (this is rough and can be simplified with a bit more refinement.)
#### cache-columns
The first thing to do is cache the columns for the project you're interested in.
This will speed up future actions because it avoids a lot of unnecessary requests
to Phabricator that would otherwise be required to resolve the names of projects
......@@ -51,13 +52,15 @@ and workboard columns.
dddcli metrics cache-columns --project=PHID-PROJ-uier7rukzszoewbhj7ja
```
Then you can fetch the actual metrics and map them into local sqlite tables:
Then you can fetch the actual metrics and map them into local sqlite tables with the map sub-command:
```bash
dddcli metrics map --project=PHID-PROJ-uier7rukzszoewbhj7ja
```
Note that `--project` accepts either a `PHID` or a project `#hashtag`, so you can try `dddcli metrics map --project=#releng`, for example.
To get cli usage help, try
```bash
......@@ -79,17 +82,42 @@ If you omit the --mock argument then it will request a rather large amount of da
To run datasette, from the ddd checkout:
```bash
dddcli serve ./www
export DATASETTE_PORT=8001
datasette --reload --metadata www/metadata.yaml -h 0.0.0.0 -p $DATASETTE_PORT www
```
Sample systemd units are in `etc/systemd/*` including a file watcher to restart datasette
when the data changes.
# Example code:
## Conduit API client:
```python
from ddd.phab import Conduit
phab = Conduit()
# Call phabricator's meniphest.search api and retrieve all results
r = phab.request('maniphest.search', {'queryKey': "KpRagEN3fCBC",
"limit": "40",
"attachments": {
"projects": True,
"columns": True
}})
```
This fetches every page of results, note the API limits a single request to
fetching **at most** 100 objects, however, fetch_all will request each page from the server until all available records have been retrieved:
```python
r.fetch_all()
```
## PHIDRef
Whenever encountering a phabricator `phid`, we use PHIDRef objects to wrap the phid. This provides several conveniences
for working with phabricator objects efficiently. This interactive python session demonstrates how it works:
Whenever encountering a phabricator `phid`, we use PHIDRef objects to wrap the phid. This provides several conveniences for working with phabricator objects efficiently. This interactive python session demonstrates how it works:
```python
In [1]: phid = PHIDRef('PHID-PROJ-uier7rukzszoewbhj7ja')
......@@ -116,31 +144,9 @@ Out[7]: PHID-PROJ-uier7rukzszoewbhj7ja
```
1. You can construct a bunch of PHIDRef instances and then later on you can fetch all of
the data in a single call to `resolve_phids()`.
2. resolve_phids can store a local cache of the phid details in the phobjects table.
3. a PHIDRef can be used transparently as a database key.
* `str(PHIDRef_instance)` returns the original `"PHID-TYPE-hash"` string.
* `PHIDRef_instance.object` returns an instantiated `PHObject` instance.
* After calling `resolve_phids()`, all `PHObject` instances will contain the `name`,
`url` and `status` of the corresponding phabricator objects.
```python
from ddd.phab import Conduit
phab = Conduit()
1. You can construct a bunch of `PHIDRef` instances and then later on you can fetch all of the data in a single call to phabricator's conduit api. This is accomplished by calling `PHObject.resolve_phids()`.
2. `resolve_phids()` can store a local cache of the phid details in the phobjects table. After calling resolve_phids completes, all `PHObject` instances will contain the `name`, `url` and `status` of the corresponding phabricator objects.
3. An instance of PHIDRef can be used transparently as a database key.
4. `str(PHIDRef_instance)` returns the original `"PHID-TYPE-hash"` string.
5. `PHIDRef_instance.object` returns an instantiated `PHObject` instance.
# Call phabricator's meniphest.search api and retrieve all results
r = phab.request('maniphest.search', {'queryKey': "KpRagEN3fCBC",
"limit": "40",
"attachments": {
"projects": True,
"columns": True
}})
# This fetches every page of results, note the API limits a single request to
# fetching at most 100 results (controlled by the limit argument)
# But fetch_all will request each page from the server until all available
# records have been retrieved.
r.fetch_all()
```
......@@ -5,7 +5,7 @@ After=network.target
[Service]
Type=simple
WorkingDirectory=/srv/ddd
ExecStart=run-datasette.sh
ExecStart=etc/ddd-datasette.sh
User=ddd
Group=srv
......
......@@ -42,4 +42,7 @@ conduit = "ddd.conduit_cli:cli"
dddcli = "ddd.main:cli"
[tool.pyright]
reportMissingTypeStubs = true
\ No newline at end of file
reportMissingTypeStubs = true
[tool.mypy]
implicit_reexport = true
\ No newline at end of file
Subproject commit bf87331e1ec2e461ceff321870ba2564a7a5829d
Subproject commit bf224aa117be3f9c882ce4de27b09f8fa3dccb8c
from rich.console import Console
console = Console(stderr=True)
......@@ -15,7 +15,7 @@ import click
from rich.console import Console
import typer
from rich.status import Status
from sqlite_utils.db import Database,chunks
from sqlite_utils.db import Database, chunks
from typer import Option, Typer
from ddd.boardmetrics_mapper import maptransactions
......@@ -38,13 +38,13 @@ def cache_tasks(conduit, cache, tasks, sts):
new_instances = []
for task in r.data:
task.save()
#instance = PHObject.instance(phid=PHID(key), data=vals, save=True)
#new_instances.append(instance)
# instance = PHObject.instance(phid=PHID(key), data=vals, save=True)
# new_instances.append(instance)
#cache.store_all(r.data)
# cache.store_all(r.data)
def cache_projects(conduit:Conduit, cache, sts):
def cache_projects(conduit: Conduit, cache, sts):
r = conduit.project_search(constraints={"maxDepth": 2})
r.fetch_all(sts)
......@@ -54,6 +54,7 @@ def cache_projects(conduit:Conduit, cache, sts):
cache.store_all(r.data)
@cli.command()
def cache_columns(ctx: typer.Context, project: str = Option("all")):
"""
......@@ -80,15 +81,13 @@ def cache_columns(ctx: typer.Context, project: str = Option("all")):
for col in r.data:
count += 1
col.save()
#col.project.save()
if round((count/total) * 100) > pct:
pct = round((count/total) * 100)
# col.project.save()
if round((count / total) * 100) > pct:
pct = round((count / total) * 100)
sts.update(
f"Saved [bold green]{count}[/bold green] ([bold blue]{pct}%[/bold blue]) Project Columns."
)
config.console.log(f"Fetched & cached {count} Project Columns.")
config.db.conn.commit()
config.console.log("Updating phobjects cache.")
......
......@@ -39,7 +39,7 @@ def maptransactions(
all_metrics = set()
@mapper("transactionType=core:edge", "meta.edge:type=41")
def projects(t, context:Task):
def projects(t, context: Task):
"""
edge transactions point to related objects such as subtasks,
mentioned tasks and project tags.
......@@ -98,26 +98,26 @@ def maptransactions(
return [("subtask_resolved", "global", t["taskID"], None)]
@mapper("transactionType=status")
def status(t, context:Task):
def status(t, context: Task):
ts = int(t["dateCreated"])
state = t["newValue"]
if state in ("open", "stalled", "progress"):
#for metric in context.metrics(is_started=False):
# for metric in context.metrics(is_started=False):
# metric.start(state)
context.metric(key='status').start(ts, state)
context.metric(key="status").start(ts, state)
elif state in ("declined", "resolved", "invalid"):
for metric in context.metrics(is_ended=False):
metric.end(ts, state)
context.metric(key='status').end(ts, state)
context.metric(key="status").end(ts, state)
return [("status", "global", t["oldValue"], t["newValue"])]
@mapper("transactionType=reassign")
def assign(t, context):
ts = int(t["dateCreated"])
if t["oldValue"]:
context.metric(key='assign').val(t['oldValue']).end(ts, 'reassign')
context.metric(key='assign').val(t['newValue']).start(ts, 'assign')
context.metric(key="assign").val(t["oldValue"]).end(ts, "reassign")
context.metric(key="assign").val(t["newValue"]).start(ts, "assign")
return [("assign", "global", t["oldValue"], t["newValue"])]
@mapper("transactionType=core:create")
......@@ -143,12 +143,14 @@ def maptransactions(
res.append(("columns", ref, fromcol, tocol))
if source or target:
for i in ('fromPHID', 'toPHID'):
for i in ("fromPHID", "toPHID"):
PHObject.instance(ref).metric(task=t["taskID"]).start(ts, tocol)
srcphid = getattr(source, i, None)
tophid = getattr(target, i, None)
if (srcphid and tophid):
PHObject.instance(ref).metric(task=t["taskID"]).start(ts, tophid)
if srcphid and tophid:
PHObject.instance(ref).metric(task=t["taskID"]).start(
ts, tophid
)
res.append(("milestone", ref, srcphid, tophid))
return res
......
......@@ -44,7 +44,9 @@ class Config:
--sql
CREATE TABLE IF NOT EXISTS events(ts, task, project phid, user phid, event, old, new);
--sql
CREATE UNIQUE INDEX IF NOT EXISTS events_pk on events(ts, task, event);
CREATE UNIQUE INDEX IF NOT EXISTS events_pk on events(ts, task, project, event, old, new);
--sql
CREATE INDEX IF NOT EXISTS events_project on events(event, project, old, new);
--sql
DROP VIEW IF EXISTS view_column_metrics;
--sql
......
......@@ -55,7 +55,7 @@ def main(
register_sqlite_adaptors()
ctx.meta["db"] = Database(db)
PHObject.db = ctx.meta["db"]
PHObject.conduit = ctx.meta['conduit']
@app.command()
def request(
......@@ -71,8 +71,8 @@ def request(
with db.conn:
for project in cursor.result["data"]:
project.save()
while 'parent' in project and project.parent:
project=project.parent
while "parent" in project and project.parent:
project = project.parent
project.save()
db.conn.commit()
......
......@@ -135,13 +135,9 @@ class Conduit(object):
"project.search", {"queryKey": queryKey, "constraints": constraints}
)
def maniphest_search(
self, constraints: MutableMapping = {}
) -> Cursor:
def maniphest_search(self, constraints: MutableMapping = {}) -> Cursor:
"""Find projects"""
return self.request(
"maniphest.search", {"constraints": constraints}
)
return self.request("maniphest.search", {"constraints": constraints})
def project_columns(
self, project: PHID = None, column_phids: Sequence = None
......
......@@ -29,7 +29,7 @@ from typing import (
)
from rich.console import Console
from sqlite_utils.db import Database,Table
from sqlite_utils.db import Database, NotFoundError, Table
console = Console()
......@@ -50,10 +50,13 @@ console = Console()
"""
class EmptyArg:
"""Sentinal Value"""
pass
class PHIDError(ValueError):
def __init__(self, msg):
self.msg = msg
......@@ -164,7 +167,10 @@ def PHIDType(phid: PHID) -> Type[PHObject]:
"""Find the class for a given PHID string. Returns a reference to the
matching subclass or to PHObject when there is no match."""
try:
parts = phid.split("-")
if isinstance(phid, PHIDRef):
parts = phid.toPHID.split("-")
else:
parts = phid.split("-")
phidtype = parts[1]
if phidtype in PHIDTypes.__members__:
classtype = PHIDTypes[phidtype].value
......@@ -258,6 +264,7 @@ class PHObject(PhabObjectBase, SubclassCache[PHID, PhabObjectBase]):
This class handles caching and insures that there is at most one instance
per unique phid.
"""
_type_name = None
@classmethod
......@@ -268,6 +275,7 @@ class PHObject(PhabObjectBase, SubclassCache[PHID, PhabObjectBase]):
return cls.__name__
db: ClassVar[Database]
conduit: ClassVar[Conduit]
table: ClassVar[Table]
savequeue: ClassVar[deque] = deque()
......@@ -314,19 +322,23 @@ class PHObject(PhabObjectBase, SubclassCache[PHID, PhabObjectBase]):
table.upsert(record, alter=True)
def load(self):
table = self.get_table()
data = table.get(self.phid)
self.update(data)
try:
table = self.get_table()
data = table.get(self.phid)
self.update(data)
except NotFoundError as e:
console.log(e)
return self
@property
def loaded(self):
def loaded(self) -> bool:
return len(self.__dict__.keys()) > 1
@classmethod
def instance(
cls, phid: PHID, data: Optional[Mapping] = None, save: bool = False
) -> PhabObjectBase:
) -> PHObject:
obj = __class__.byid(phid)
if not obj:
phidtype = PHIDType(phid)
......@@ -338,10 +350,10 @@ class PHObject(PhabObjectBase, SubclassCache[PHID, PhabObjectBase]):
obj.update(data)
if save:
obj.save()
return obj
return obj #type:ignore
@classmethod
def resolve_phids(cls, conduit, cache: Optional[DataCache] = None):
def resolve_phids(cls, conduit: Optional[Conduit] = None, cache: Optional[DataCache] = None):
phids = {phid: True for phid in __class__.instances.keys()}
if cache:
......@@ -353,6 +365,8 @@ class PHObject(PhabObjectBase, SubclassCache[PHID, PhabObjectBase]):
# no more phids to resolve.
return cls.instances
phids = [phid for phid in phids.keys()]
if not conduit:
conduit = PHObject.conduit
res = conduit.raw_request(method="phid.query", args={"phids": phids})
objs = res.json()
......@@ -466,7 +480,7 @@ class TimeSpanMetric:
@task.setter
def task(self, val):
if isinstance(val, str):
val = int(val[1:]) if val[0] == 'T' else int(val)
val = int(val[1:]) if val[0] == "T" else int(val)
self._task = val
def __repr__(self) -> str:
......@@ -475,7 +489,7 @@ class TimeSpanMetric:
class MetricsMixin:
_metrics: defaultdict[Any, TimeSpanMetric]
phid:PHID
phid: PHID
def metrics(self, is_started=True, is_ended=True):
"""Retrieve metrics, optionally filtered by their state"""
......@@ -492,10 +506,10 @@ class MetricsMixin:
if m.last is not None:
m.end(ts, state)
def metric(self, metric:TimeSpanMetric=None, **kwds) -> TimeSpanMetric:
def metric(self, metric: TimeSpanMetric = None, **kwds) -> TimeSpanMetric:
"""Create a metric instance for a given task+project pair"""
if "key" in kwds and kwds['key'] is not None:
key = kwds['key']
if "key" in kwds and kwds["key"] is not None:
key = kwds["key"]
else:
key = self.phid
......@@ -551,7 +565,7 @@ class Project(PHObjectWithFields, MetricsMixin):
class ProjectColumn(PHObjectWithFields):
_type_name = 'Column'
_type_name = "Column"
derive_columns = DerivedTable(
"columns",
......@@ -569,12 +583,13 @@ class ProjectColumn(PHObjectWithFields):
class Task(PHObjectWithFields, MetricsMixin):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._metrics = defaultdict(TimeSpanMetric)
def metric(self, key:str, metric: Optional[TimeSpanMetric]=None) -> TimeSpanMetric:
def metric(
self, key: str, metric: Optional[TimeSpanMetric] = None
) -> TimeSpanMetric:
return super().metric(key=key, metric=metric, task=self.phid)
......@@ -825,9 +840,9 @@ class PHObjectEncoder(json.JSONEncoder):
return json.JSONEncoder.default(self, o)
class NotFound:
"""Sentinal value"""
pass
......@@ -906,9 +921,9 @@ class KKVCache(object):
elif ptype is Project:
res = self.get_default_column(phid)
#if res is None:
# if res is None:
# res = PHIDRef(phid)
#console.log(repr(res))
# console.log(repr(res))
return res
......@@ -952,7 +967,7 @@ class DataCache(object):
sql = f"""{op} INTO {self.table_name} ({", ".join(cols)}) VALUES ({self.placeholders(len(cols))})"""
return sql
def __init__(self, db: Database|sqlite3.Connection, create_schema=True):
def __init__(self, db: Database | sqlite3.Connection, create_schema=True):
if isinstance(db, Database):
self.con = db.conn
......@@ -1011,6 +1026,7 @@ class DataCache(object):
def store_one(self, item: PHObject):
self.store_all([item])
KKV: KKVCache
DATA: DataCache
......
#!/usr/bin/python3
from operator import itemgetter
from pprint import pprint
from typing import List
from typing import Any, List
import pandas as pd
import typer
from sqlite_utils import Database
from sqlite_utils.db import Database
from ddd.mw import MWVersion, version
from ddd.phab import Conduit
......@@ -26,7 +26,8 @@ def blockers(
constraints = {
"subtypes": ["release"],
}
} # type: dict[str, list[Any]]
if len(task_ids) and task_ids[0] != "all":
constraints["ids"] = [int(id.strip(" T")) for id in task_ids]
......@@ -119,7 +120,7 @@ def blockers(
# elif metric == 'comment':
# train_comments.upsert([])
pprint(context)
# pprint(context)
def main():
......
......@@ -2,6 +2,8 @@
This type stub file was generated by pyright.
"""
from datasette.version import __version__, __version_info__
from .hookspecs import hookimpl, hookspec
from datasette.version import __version__ as __version__
from datasette.version __version_info__ as __version_info__
from .hookspecs import hookimpl as hookimpl
from .hookspecs import hookspec as hookspec
......@@ -2,6 +2,7 @@
This type stub file was generated by pyright.
"""
from datasette.database import Database
from .utils.asgi import NotFound
app_root = ...
......@@ -18,110 +19,110 @@ class Datasette:
ERROR = ...
def __init__(self, files, immutables=..., cache_headers=..., cors=..., inspect_data=..., metadata=..., sqlite_extensions=..., template_dir=..., plugins_dir=..., static_mounts=..., memory=..., config=..., secret=..., version_note=..., config_dir=..., pdb=..., crossdb=...) -> None:
...
async def refresh_schemas(self): # -> None:
...
@property
def urls(self): # -> Urls:
...
async def invoke_startup(self): # -> None:
...
def sign(self, value, namespace=...): # -> _t_str_bytes:
...
def unsign(self, signed, namespace=...): # -> Any:
...
def get_database(self, name=...):
def get_database(self, name=...) -> Database:
...
def add_database(self, db, name=...):
...
def add_memory_database(self, memory_name): # -> Database:
def add_memory_database(self, memory_name) -> Database:
...
def remove_database(self, name): # -> None:
...