Experiment Draft Checklist is updated after running validation
Adding a new feature to the poller to validate all experiments when they are still a Draft. Every validation rule that belongs to a Pre-Launch Checklist item is validated and the corresponding item is checked if passed:
growthBookPoller: has been refactored a bit. Now all experiments are fetched to be able to validate the running as before and also the drafts according to this new featuregrowthBookClient:- A new function has been added,
completeExperimentChecklistto mark as checked all the Checklist items of a specific Draft for those rules that have passed the specific validation process for drafts - Also,
fetchExperimentshas been refactored tofetchAllExperimentsto fetch all experiments. That way the poller can run validation for the ones that arerunningalready and also for the ones that are still adraft - Took the opportunity to remove the nested
experimentsproperty from the response that GrowthBook send when fetching the experiments viafetchAllExperiments. Now the poller deals directly with an array of GB experiments which is simpler
- A new function has been added,
growthBookUtils: Added the necessary to be able to map from a specific Validation Error or Warning message, found during the validation process, to the related Checklist Item. That way we know which related item needs to be mark as checked when a specific validation rule is passed- Added a new config property,
growthbook.draft_validation: true | falseto enable/disable experiment draft validations. We still have to deal with API rate limits and, although we have some ideas we can work on, it would be good to be able to disable this new feature just in case - Added a new config property,
growthbook.draft_validation_interval: numberto set how often, relative to the poller, drafts validation/Checklist update will happen (with4as the default value)
Also:
- Took the opportunity to remove
batchSlugs. GrowthBook is already checking that two experiments cannot have the sametrackingKeyso that feature is not needed and we can simplify things a bit
Initially we planned to uncheck also those Checklist items that don't pass the validation process but I found that GrowthBook API doesn't provide any endpoint to do that, so Test Kitchen is only able to check an item after being validated but that item won't be unchecked if the validation rule is broken after that.
Bug: T429420