🏭 Mega Scale Multimodal DataPipeline for SOTA Foundation Models
Official Implementation of "CLI-Gym: Scalable CLI Task Generation via Agentic Environment Inversion"
Simple stream processing pipeline
Build super simple end-to-end data & ETL pipelines for your vector databases and Generative AI applications
High Performance Tensorflow Data Pipeline with State of Art Augmentations and low level optimizations.
kedro cli plugin for generating a static kedro viz site (html, css, js) that can be deployed on many serverless tools.
DBT and clickhouse test project with dagster
Modeling tool like DBT to use SQL Alchemy core with a DataFrame interface like
Extract transform load CLI tool for extracting small and middle data volume from sources (databases, csv files, xls files, gspreadsheets) to target (databases, csv files, xls files, gspreadsheets) in free combination.
Материалы для курса Введение в Data Engineering: дата пайплайны
Simple Airflow on Kubernetes (GKE)
This is an ETL project - extracting data from an ecommerce transactional database on RDS, transforming the data using AWS glue job, and loading it to a Redshift data warehouse, and connected it to Tableau for BI
MustRD: Validate your SPARQL queries and transformations with precision and confidence, using BDD and Given-When-Then principles
An ETL data pipeline that extracts data from source and loads it to destination, automated using mage.ai
Packages for the Tenzir ecosystem.
This is a project which demonstrates creation of a data pipeline by scraping data using twitter API and creating a data delivery stream using Kinesis Firehose for ingesting data to Amazon S3.
Singer is a complete training and evaluation pipeline for classical Chinese regulated verse (格律诗).
To associate your repository with the datapipeline topic, visit your repo's landing page and select "manage topics."