Experiment Draft Checklist is updated after running validation

Adding a new feature to the poller to validate all experiments when they are still a Draft. Every validation rule that belongs to a Pre-Launch Checklist item is validated and the corresponding item is checked if passed:

  • growthBookPoller : has been refactored a bit. Now all experiments are fetched to be able to validate the running as before and also the drafts according to this new feature
  • growthBookClient :
    • A new function has been added, completeExperimentChecklist to mark as checked all the Checklist items of a specific Draft for those rules that have passed the specific validation process for drafts
    • Also, fetchExperiments has been refactored to fetchAllExperiments to fetch all experiments. That way the poller can run validation for the ones that are running already and also for the ones that are still a draft
    • Took the opportunity to remove the nested experiments property from the response that GrowthBook send when fetching the experiments via fetchAllExperiments. Now the poller deals directly with an array of GB experiments which is simpler
  • growthBookUtils: Added the necessary to be able to map from a specific Validation Error or Warning message, found during the validation process, to the related Checklist Item. That way we know which related item needs to be mark as checked when a specific validation rule is passed
  • Added a new config property, growthbook.draft_validation: true | false to enable/disable experiment draft validations. We still have to deal with API rate limits and, although we have some ideas we can work on, it would be good to be able to disable this new feature just in case
  • Added a new config property, growthbook.draft_validation_interval: number to set how often, relative to the poller, drafts validation/Checklist update will happen (with 4 as the default value)

Also:

  • Took the opportunity to remove batchSlugs. GrowthBook is already checking that two experiments cannot have the same trackingKey so that feature is not needed and we can simplify things a bit

Initially we planned to uncheck also those Checklist items that don't pass the validation process but I found that GrowthBook API doesn't provide any endpoint to do that, so Test Kitchen is only able to check an item after being validated but that item won't be unchecked if the validation rule is broken after that.

Bug: T429420

Edited by Santiago Faci

Merge request reports

Loading