Jinja dashboards

.gitignore

 __pycache__
 .mypy_cache
+.pytest_cache
 *.csv
 *.pyc
 .ipynb_checkpoints
 .spyproject
 .vscode
-cache.db
 *.egg-info
+*.db
 /.eggs
 /dist
-*.db
-/.venv/*
+/.venv
 /test/*.json
 /ddd.code-workspace
 /ignore/*

README.md

@@ -6,7 +6,7 @@
 ## Status
 
 This tool an supporting libraries are in early stages of experimentation and
-development. The APIs are not yet stable and the featureset is not yet decoded
+development. The APIs are not yet stable and the featureset is not yet decided
 let alone completely implemented. Stay tuned or get involved.
 
 ## Currently supported data sources:
@@ -21,7 +21,7 @@ let alone completely implemented. Stay tuned or get involved.
 
 # Usage
 
-## cli
+## Installation
 
 setup.py will install a command line tool called `dddcli`
 
@@ -31,17 +31,18 @@ To install for development use:
 python3 setup.py develop
 ```
 
+### dddcli
 
-You can use the following sub-commands with `dddcli command [args]` to access various functionality.
+You can use the following sub-commands by running `dddcli sub-command [args]` to access various functionality.
 
-### phabricator metrics
+### Phabricator metrics:  `dddcli metrics`
 
 This tool is used to extract data from phabricator and organize it in a structure that will facilitate further analysis.
 The analysis of task activities can provide some insight into workflows.
 The output if this tool will be used as the data source for charts to visualize certain agile project planning metrics.
 
-Example usage (this is rough and can be simplified with a bit more refinement.)
 
+#### cache-columns
 The first thing to do is cache the columns for the project you're interested in.
 This will speed up future actions because it avoids a lot of unnecessary requests
 to Phabricator that would otherwise be required to resolve the names of projects
@@ -51,13 +52,15 @@ and workboard columns.
 dddcli metrics cache-columns --project=PHID-PROJ-uier7rukzszoewbhj7ja
 ```
 
-Then you can fetch the actual metrics and map them into local sqlite tables:
+Then you can fetch the actual metrics and map them into local sqlite tables with the map sub-command:
 
 
 ```bash
 dddcli metrics map --project=PHID-PROJ-uier7rukzszoewbhj7ja
 ```
 
+Note that `--project` accepts either a `PHID` or a project `#hashtag`, so you can try `dddcli metrics map --project=#releng`, for example.
+
 To get cli usage help, try
 
 ```bash
@@ -79,17 +82,42 @@ If you omit the --mock argument then it will request a rather large amount of da
 To run datasette, from the ddd checkout:
 
 ```bash
-dddcli serve ./www
+export DATASETTE_PORT=8001
+datasette --reload --metadata www/metadata.yaml -h 0.0.0.0 -p $DATASETTE_PORT  www
 ```
+
 Sample systemd units are in `etc/systemd/*` including a file watcher to restart datasette
 when the data changes.
 
 # Example code:
 
+## Conduit API client:
+
+```python
+from ddd.phab import Conduit
+
+phab = Conduit()
+
+# Call phabricator's meniphest.search api and retrieve all results
+r = phab.request('maniphest.search', {'queryKey': "KpRagEN3fCBC",
+                "limit": "40",
+                "attachments": {
+                    "projects": True,
+                    "columns": True
+                }})
+```
+
+This fetches every page of results, note the API limits a single request to
+fetching **at most** 100 objects, however, fetch_all will request each page from the server until all available records have been retrieved:
+
+```python
+r.fetch_all()
+```
+
+
 ## PHIDRef
 
-Whenever encountering a phabricator `phid`, we use PHIDRef objects to wrap the phid. This provides several conveniences
-for working with phabricator objects efficiently.  This interactive python session demonstrates how it works:
+Whenever encountering a phabricator `phid`, we use PHIDRef objects to wrap the phid. This provides several conveniences for working with phabricator objects efficiently.  This interactive python session demonstrates how it works:
 
 ```python
 In [1]: phid = PHIDRef('PHID-PROJ-uier7rukzszoewbhj7ja')
@@ -116,31 +144,9 @@ Out[7]: PHID-PROJ-uier7rukzszoewbhj7ja
 
 ```
 
-1. You can construct a bunch of PHIDRef instances and then later on you can fetch all of
-   the data in a single call to `resolve_phids()`.
-2. resolve_phids can store a local cache of the phid details in the phobjects table.
-3. a PHIDRef can be used transparently as a database key.
- * `str(PHIDRef_instance)` returns the original `"PHID-TYPE-hash"` string.
- * `PHIDRef_instance.object` returns an instantiated `PHObject` instance.
- * After calling `resolve_phids()`, all `PHObject` instances will contain the `name`,
-   `url` and `status` of the corresponding phabricator objects.
-
-
-```python
-from ddd.phab import Conduit
-
-phab = Conduit()
+1. You can construct a bunch of `PHIDRef` instances and then later on you can fetch all of the data in a single call to phabricator's conduit api. This is accomplished by calling `PHObject.resolve_phids()`.
+2. `resolve_phids()` can store a local cache of the phid details in the phobjects table. After calling resolve_phids completes, all `PHObject` instances will contain the `name`, `url` and `status` of the corresponding phabricator objects.
+3. An instance of PHIDRef can be used transparently as a database key.
+4. `str(PHIDRef_instance)` returns the original `"PHID-TYPE-hash"` string.
+5. `PHIDRef_instance.object` returns an instantiated `PHObject` instance.
 
-# Call phabricator's meniphest.search api and retrieve all results
-r = phab.request('maniphest.search', {'queryKey': "KpRagEN3fCBC",
-                "limit": "40",
-                "attachments": {
-                    "projects": True,
-                    "columns": True
-                }})
-# This fetches every page of results, note the API limits a single request to
-# fetching at most 100 results (controlled by the limit argument)
-# But fetch_all will request each page from the server until all available
-# records have been retrieved.
-r.fetch_all()
-```

etc/systemd/ddd-datasette.service

@@ -5,7 +5,7 @@ After=network.target
 [Service]
 Type=simple
 WorkingDirectory=/srv/ddd
-ExecStart=run-datasette.sh
+ExecStart=etc/ddd-datasette.sh
 User=ddd
 Group=srv
 

pyproject.toml

@@ -43,3 +43,6 @@ dddcli = "ddd.main:cli"
 
 [tool.pyright]
 reportMissingTypeStubs = true
+
+[tool.mypy]
+implicit_reexport = true
\ No newline at end of file

datasette-dashboards @ bf224aa1

-Subproject commit bf87331e1ec2e461ceff321870ba2564a7a5829d
+Subproject commit bf224aa117be3f9c882ce4de27b09f8fa3dccb8c

src/ddd/__init__.py

+
+from rich.console import Console
+console = Console(stderr=True)

src/ddd/boardmetrics.py

@@ -15,7 +15,7 @@ import click
 from rich.console import Console
 import typer
 from rich.status import Status
-from sqlite_utils.db import Database,chunks
+from sqlite_utils.db import Database, chunks
 from typer import Option, Typer
 
 from ddd.boardmetrics_mapper import maptransactions
@@ -38,13 +38,13 @@ def cache_tasks(conduit, cache, tasks, sts):
     new_instances = []
     for task in r.data:
         task.save()
-        #instance = PHObject.instance(phid=PHID(key), data=vals, save=True)
-        #new_instances.append(instance)
+        # instance = PHObject.instance(phid=PHID(key), data=vals, save=True)
+        # new_instances.append(instance)
 
-    #cache.store_all(r.data)
+    # cache.store_all(r.data)
 
 
-def cache_projects(conduit:Conduit, cache, sts):
+def cache_projects(conduit: Conduit, cache, sts):
     r = conduit.project_search(constraints={"maxDepth": 2})
 
     r.fetch_all(sts)
@@ -54,6 +54,7 @@ def cache_projects(conduit:Conduit, cache, sts):
 
     cache.store_all(r.data)
 
+
 @cli.command()
 def cache_columns(ctx: typer.Context, project: str = Option("all")):
     """
@@ -80,15 +81,13 @@ def cache_columns(ctx: typer.Context, project: str = Option("all")):
         for col in r.data:
             count += 1
             col.save()
-            #col.project.save()
-            if round((count/total) * 100) > pct:
-                pct = round((count/total) * 100)
+            # col.project.save()
+            if round((count / total) * 100) > pct:
+                pct = round((count / total) * 100)
                 sts.update(
                     f"Saved [bold green]{count}[/bold green] ([bold blue]{pct}%[/bold blue]) Project Columns."
                 )
 
-
-
     config.console.log(f"Fetched & cached {count} Project Columns.")
     config.db.conn.commit()
     config.console.log("Updating phobjects cache.")

src/ddd/boardmetrics_mapper.py

@@ -39,7 +39,7 @@ def maptransactions(
     all_metrics = set()
 
     @mapper("transactionType=core:edge", "meta.edge:type=41")
-    def projects(t, context:Task):
+    def projects(t, context: Task):
         """
         edge transactions point to related objects such as subtasks,
         mentioned tasks and project tags.
@@ -98,26 +98,26 @@ def maptransactions(
         return [("subtask_resolved", "global", t["taskID"], None)]
 
     @mapper("transactionType=status")
-    def status(t, context:Task):
+    def status(t, context: Task):
         ts = int(t["dateCreated"])
 
         state = t["newValue"]
         if state in ("open", "stalled", "progress"):
-            #for metric in context.metrics(is_started=False):
+            # for metric in context.metrics(is_started=False):
             #    metric.start(state)
-            context.metric(key='status').start(ts, state)
+            context.metric(key="status").start(ts, state)
         elif state in ("declined", "resolved", "invalid"):
             for metric in context.metrics(is_ended=False):
                 metric.end(ts, state)
-            context.metric(key='status').end(ts, state)
+            context.metric(key="status").end(ts, state)
         return [("status", "global", t["oldValue"], t["newValue"])]
 
     @mapper("transactionType=reassign")
     def assign(t, context):
         ts = int(t["dateCreated"])
         if t["oldValue"]:
-            context.metric(key='assign').val(t['oldValue']).end(ts, 'reassign')
-        context.metric(key='assign').val(t['newValue']).start(ts, 'assign')
+            context.metric(key="assign").val(t["oldValue"]).end(ts, "reassign")
+        context.metric(key="assign").val(t["newValue"]).start(ts, "assign")
         return [("assign", "global", t["oldValue"], t["newValue"])]
 
     @mapper("transactionType=core:create")
@@ -143,12 +143,14 @@ def maptransactions(
                 res.append(("columns", ref, fromcol, tocol))
 
                 if source or target:
-                    for i in ('fromPHID', 'toPHID'):
+                    for i in ("fromPHID", "toPHID"):
                         PHObject.instance(ref).metric(task=t["taskID"]).start(ts, tocol)
                         srcphid = getattr(source, i, None)
                         tophid = getattr(target, i, None)
-                        if (srcphid and tophid):
-                            PHObject.instance(ref).metric(task=t["taskID"]).start(ts, tophid)
+                        if srcphid and tophid:
+                            PHObject.instance(ref).metric(task=t["taskID"]).start(
+                                ts, tophid
+                            )
                             res.append(("milestone", ref, srcphid, tophid))
         return res
 

src/ddd/boardmetrics_schema.py

@@ -44,7 +44,9 @@ class Config:
             --sql
             CREATE TABLE IF NOT EXISTS events(ts, task, project phid, user phid, event, old, new);
             --sql
-            CREATE UNIQUE INDEX IF NOT EXISTS events_pk on events(ts, task, event);
+            CREATE UNIQUE INDEX IF NOT EXISTS events_pk on events(ts, task, project, event, old, new);
+            --sql
+            CREATE INDEX IF NOT EXISTS events_project on events(event, project, old, new);
             --sql
             DROP VIEW IF EXISTS view_column_metrics;
             --sql

src/ddd/main.py

@@ -55,7 +55,7 @@ def main(
     register_sqlite_adaptors()
     ctx.meta["db"] = Database(db)
     PHObject.db = ctx.meta["db"]
-
+    PHObject.conduit = ctx.meta['conduit']
 
 @app.command()
 def request(
@@ -71,8 +71,8 @@ def request(
     with db.conn:
         for project in cursor.result["data"]:
             project.save()
-            while 'parent' in project and project.parent:
-                project=project.parent
+            while "parent" in project and project.parent:
+                project = project.parent
                 project.save()
     db.conn.commit()
 

src/ddd/phab.py

@@ -135,13 +135,9 @@ class Conduit(object):
             "project.search", {"queryKey": queryKey, "constraints": constraints}
         )
 
-    def maniphest_search(
-        self, constraints: MutableMapping = {}
-    ) -> Cursor:
+    def maniphest_search(self, constraints: MutableMapping = {}) -> Cursor:
         """Find projects"""
-        return self.request(
-            "maniphest.search", {"constraints": constraints}
-        )
+        return self.request("maniphest.search", {"constraints": constraints})
 
     def project_columns(
         self, project: PHID = None, column_phids: Sequence = None

src/ddd/phobjects.py

@@ -29,7 +29,7 @@ from typing import (
 )
 
 from rich.console import Console
-from sqlite_utils.db import Database,Table
+from sqlite_utils.db import Database, NotFoundError, Table
 
 console = Console()
 
@@ -50,10 +50,13 @@ console = Console()
 
  """
 
+
 class EmptyArg:
     """Sentinal Value"""
+
     pass
 
+
 class PHIDError(ValueError):
     def __init__(self, msg):
         self.msg = msg
@@ -164,6 +167,9 @@ def PHIDType(phid: PHID) -> Type[PHObject]:
     """Find the class for a given PHID string. Returns a reference to the
     matching subclass or to PHObject when there is no match."""
     try:
+        if isinstance(phid, PHIDRef):
+            parts = phid.toPHID.split("-")
+        else:
             parts = phid.split("-")
         phidtype = parts[1]
         if phidtype in PHIDTypes.__members__:
@@ -258,6 +264,7 @@ class PHObject(PhabObjectBase, SubclassCache[PHID, PhabObjectBase]):
     This class handles caching and insures that there is at most one instance
     per unique phid.
     """
+
     _type_name = None
 
     @classmethod
@@ -268,6 +275,7 @@ class PHObject(PhabObjectBase, SubclassCache[PHID, PhabObjectBase]):
             return cls.__name__
 
     db: ClassVar[Database]
+    conduit: ClassVar[Conduit]
     table: ClassVar[Table]
     savequeue: ClassVar[deque] = deque()
 
@@ -314,19 +322,23 @@ class PHObject(PhabObjectBase, SubclassCache[PHID, PhabObjectBase]):
             table.upsert(record, alter=True)
 
     def load(self):
+        try:
             table = self.get_table()
             data = table.get(self.phid)
             self.update(data)
+        except NotFoundError as e:
+            console.log(e)
+
         return self
 
     @property
-    def loaded(self):
+    def loaded(self) -> bool:
         return len(self.__dict__.keys()) > 1
 
     @classmethod
     def instance(
         cls, phid: PHID, data: Optional[Mapping] = None, save: bool = False
-    ) -> PhabObjectBase:
+    ) -> PHObject:
         obj = __class__.byid(phid)
         if not obj:
             phidtype = PHIDType(phid)
@@ -338,10 +350,10 @@ class PHObject(PhabObjectBase, SubclassCache[PHID, PhabObjectBase]):
             obj.update(data)
             if save:
                 obj.save()
-        return obj
+        return obj #type:ignore
 
     @classmethod
-    def resolve_phids(cls, conduit, cache: Optional[DataCache] = None):
+    def resolve_phids(cls, conduit: Optional[Conduit] = None, cache: Optional[DataCache] = None):
         phids = {phid: True for phid in __class__.instances.keys()}
 
         if cache:
@@ -353,6 +365,8 @@ class PHObject(PhabObjectBase, SubclassCache[PHID, PhabObjectBase]):
             # no more phids to resolve.
             return cls.instances
         phids = [phid for phid in phids.keys()]
+        if not conduit:
+            conduit = PHObject.conduit
         res = conduit.raw_request(method="phid.query", args={"phids": phids})
 
         objs = res.json()
@@ -466,7 +480,7 @@ class TimeSpanMetric:
     @task.setter
     def task(self, val):
         if isinstance(val, str):
-            val = int(val[1:]) if val[0] == 'T' else int(val)
+            val = int(val[1:]) if val[0] == "T" else int(val)
         self._task = val
 
     def __repr__(self) -> str:
@@ -475,7 +489,7 @@ class TimeSpanMetric:
 
 class MetricsMixin:
     _metrics: defaultdict[Any, TimeSpanMetric]
-    phid:PHID
+    phid: PHID
 
     def metrics(self, is_started=True, is_ended=True):
         """Retrieve metrics, optionally filtered by their state"""
@@ -492,10 +506,10 @@ class MetricsMixin:
             if m.last is not None:
                 m.end(ts, state)
 
-    def metric(self, metric:TimeSpanMetric=None, **kwds) -> TimeSpanMetric:
+    def metric(self, metric: TimeSpanMetric = None, **kwds) -> TimeSpanMetric:
         """Create a metric instance for a given task+project pair"""
-        if "key" in kwds and kwds['key'] is not None:
-            key = kwds['key']
+        if "key" in kwds and kwds["key"] is not None:
+            key = kwds["key"]
         else:
             key = self.phid
 
@@ -551,7 +565,7 @@ class Project(PHObjectWithFields, MetricsMixin):
 
 
 class ProjectColumn(PHObjectWithFields):
-    _type_name = 'Column'
+    _type_name = "Column"
 
     derive_columns = DerivedTable(
         "columns",
@@ -569,12 +583,13 @@ class ProjectColumn(PHObjectWithFields):
 
 
 class Task(PHObjectWithFields, MetricsMixin):
-
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self._metrics = defaultdict(TimeSpanMetric)
 
-    def metric(self, key:str, metric: Optional[TimeSpanMetric]=None) -> TimeSpanMetric:
+    def metric(
+        self, key: str, metric: Optional[TimeSpanMetric] = None
+    ) -> TimeSpanMetric:
         return super().metric(key=key, metric=metric, task=self.phid)
 
 
@@ -825,9 +840,9 @@ class PHObjectEncoder(json.JSONEncoder):
         return json.JSONEncoder.default(self, o)
 
 
-
 class NotFound:
     """Sentinal value"""
+
     pass
 
 
@@ -906,9 +921,9 @@ class KKVCache(object):
         elif ptype is Project:
             res = self.get_default_column(phid)
 
-        #if res is None:
+        # if res is None:
         #    res = PHIDRef(phid)
-        #console.log(repr(res))
+        # console.log(repr(res))
         return res
 
 
@@ -952,7 +967,7 @@ class DataCache(object):
         sql = f"""{op} INTO {self.table_name} ({", ".join(cols)}) VALUES ({self.placeholders(len(cols))})"""
         return sql
 
-    def __init__(self, db: Database|sqlite3.Connection, create_schema=True):
+    def __init__(self, db: Database | sqlite3.Connection, create_schema=True):
 
         if isinstance(db, Database):
             self.con = db.conn
@@ -1011,6 +1026,7 @@ class DataCache(object):
     def store_one(self, item: PHObject):
         self.store_all([item])
 
+
 KKV: KKVCache
 DATA: DataCache
 

src/ddd/trainblockers.py

 #!/usr/bin/python3
 from operator import itemgetter
 from pprint import pprint
-from typing import List
+from typing import Any, List
 
 import pandas as pd
 import typer
-from sqlite_utils import Database
+from sqlite_utils.db import Database
 
 from ddd.mw import MWVersion, version
 from ddd.phab import Conduit
@@ -26,7 +26,8 @@ def blockers(
 
     constraints = {
         "subtypes": ["release"],
-    }
+    }  # type: dict[str, list[Any]]
+
     if len(task_ids) and task_ids[0] != "all":
         constraints["ids"] = [int(id.strip(" T")) for id in task_ids]
 
@@ -119,7 +120,7 @@ def blockers(
             # elif metric == 'comment':
             #    train_comments.upsert([])
 
-    pprint(context)
+    # pprint(context)
 
 
 def main():

typings/datasette/__init__.pyi

@@ -2,6 +2,8 @@
 This type stub file was generated by pyright.
 """
 
-from datasette.version import __version__, __version_info__
-from .hookspecs import hookimpl, hookspec
+from datasette.version import __version__ as __version__
+from datasette.version __version_info__ as __version_info__
+from .hookspecs import hookimpl as hookimpl
+from .hookspecs import hookspec as hookspec
 

typings/datasette/app.pyi

@@ -2,6 +2,7 @@
 This type stub file was generated by pyright.
 """
 
+from datasette.database import Database
 from .utils.asgi import NotFound
 
 app_root = ...
@@ -35,13 +36,13 @@ class Datasette:
     def unsign(self, signed, namespace=...): # -> Any:
         ...
 
-    def get_database(self, name=...):
+    def get_database(self, name=...) -> Database:
         ...
 
     def add_database(self, db, name=...):
         ...
 
-    def add_memory_database(self, memory_name): # -> Database:
+    def add_memory_database(self, memory_name) -> Database:
         ...
 
     def remove_database(self, name): # -> None:

www/plugins/ddd_datasette.py

-import ddd
+from pathlib import Path
+
+from ddd import console
 from ddd.phab import Conduit
 import inspect
 from pprint import pprint
 from typing import Dict, NewType, Type, TypeVar
 from ddd.phobjects import PHID, PHIDRef, PHObject
-from datasette import hookimpl
+from datasette.hookspecs import hookimpl
 from importlib import import_module
-from datasette.app import Datasette, Database
+from datasette.app import Datasette
+from datasette.database import Database
 
 Query = NewType('Query', str)
 
@@ -134,10 +137,11 @@ class MetricsQueryList(QueryList, database='metrics'):
             count(distinct task) as tasks,
             count(distinct user) as people,
             count(distinct old) as sources,
+            old as source,
             new as column
         FROM events
         WHERE event in ('columns', 'projects') and project=:project