Skip to content
GitLab
Projects Groups Snippets
  • /
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in
  • Airflow DAGs Airflow DAGs
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Custom issue tracker
    • Custom issue tracker
  • Merge requests 7
    • Merge requests 7
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Packages and registries
    • Packages and registries
    • Package Registry
    • Infrastructure Registry
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Activity
  • Graph
  • Jobs
  • Commits
Collapse sidebar
  • repos
  • data-engineering
  • Airflow DAGsAirflow DAGs
  • Merge requests
  • !196

Draft: Add Jupyter notebook operator

  • Review changes

  • Download
  • Email patches
  • Plain diff
Open Mforns requested to merge notebooks into main Dec 14, 2022
  • Overview 0
  • Commits 4
  • Pipelines 1
  • Changes 2

This MR is the result of the spike to test the execution of Jupyter notebooks in Airflow using Skein and Papermill. It implements a NotebookOperator (inheriting from SimpleSkeinOperator) that executes a given notebook. The operator's parameters are the notebook itself, a packaged conda env to use as an environment for Skein, and the parameters for the notebook. This MR also implements a test DAG that uses the NotebookOperator.

This code is not final, it is intended to be reused when we prioritize implementing this functionality.

Assignee
Assign to
Reviewers
Request review from
Time tracking
Source branch: notebooks