Data Cleaning

pathway

by pathwaycom

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.

rocketride-server

by rocketride-org

High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent…

cocoindex

by cocoindex-io

Incremental engine for long horizon agents 🌟 Star if you like it!

cleanlab

by cleanlab

Cleanlab's open-source library is the standard data-centric AI package for data quality and machine learning with messy, real-world data and labels.

fiftyone

by voxel51

Refine high-quality datasets and visual AI models

Bash-Oneliner

by onceupon

A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.

miller

by johnkerl

Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON

DataFlow

by OpenDCAI

[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.

dasel

by TomWright

Unified querying, transformation, and modification of JSON, TOML, YAML, XML, INI, HCL, KDL and CSV.

data-juicer

by datajuicer

Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷

DALI

by NVIDIA

A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.

smallpond

by deepseek-ai

A lightweight data processing framework built on DuckDB and 3FS.

pandera

by unionai-oss

A light-weight, flexible, and expressive statistical data testing library

numaflow

by numaproj

Kubernetes-native platform to run massively parallel data/streaming jobs

datachain

by datachain-ai

The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure

broadway

by elixir-broadway

Concurrent and multi-stage data ingestion and data processing with Elixir

texar

by asyml

Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/

pandas-videos

by justmarkham

Jupyter notebook and datasets from the pandas video series

bytewax

by bytewax

Python Stream Processing

Curator

by NVIDIA-NeMo

Scalable data pre processing and curation toolkit for LLMs

skrub

by skrub-data

Machine learning with dataframes

DAT8

by justmarkham

General Assembly's 2015 Data Science course in Washington, DC

bonobo

by python-bonobo

Extract Transform Load for Python 3.5+

dolma

by allenai

Data and tools for generating and inspecting OLMo pre-training data.

optimus

by hi-primus

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

pyper

by pyper-dev

Concurrent Python made simple

janitor

by sfirke

simple tools for data cleaning in R

data-forge-ts

by data-forge

The JavaScript data transformation and analysis toolkit inspired by Pandas and LINQ.

kubetorch

by run-house

Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.

sql-data-warehouse-project

by DataWithBaraa

A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.

EduChat

by ECNU-ICALK

An open-source educational chat model from ICALK, East China Normal University. 开源中英教育对话大模型。(通用基座模型，GPU部署，数据清理) 致敬: LLaMA, MOSS, BELLE, Ziya, vLLM

xidel

by benibela

Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or…

awesome-data-agents

by HKUSTDial

Continuously updated paper list on advancements in Data Agents. Companion repo to our paper "A Survey of Data Agents: Emerging Paradigm or Overstated Hype?"

text-dedup

by ChenghaoMou

All-in-one text de-duplication

texar-pytorch

by asyml

Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/

Units_of_Measure_Harmonization-intelligence-platform

by JulietMirambo

Production-Grade ML System for Automated Unit of Measure Error Detection | 88-92% Accuracy | 94% Autonomy | KNIME Workflow

hstream

by hstreamdb

HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications.

python-nameparser

by derek73

Parse human names into title, given, middle, family, suffix, nickname and maiden name. Typed, zero-dependency, CJK-aware.

collapse

by fastverse

Advanced and Fast Data Transformation in R

awesome-kafka

by infoslack

A list about Apache Kafka

eternal

by kousun12

👾~ music, eternal ~ 👾

Machine-Learning-Notebooks

by maykulkarni

Machine Learning notebooks for refreshing concepts.

synthBTC

by jofpin

A tool that uses advanced Monte Carlo simulations and Turbit parallel processing to create possible Bitcoin prediction scenarios.

klib

by akanz1

Easy to use Python library of customized functions for cleaning and analyzing data.

tangle

by TangleML

Tangle is a web app that allows the users to build and run Machine Learning pipelines without having to set up development environment.

desbordante-core

by Desbordante

Desbordante is a high-performance data profiler that is capable of discovering many different patterns in data using various algorithms. It also allows to run data cleaning…

Musoq

by Puchaczov

SQL Runtime without any database

schema-inspector

by schema-inspector

Schema-Inspector is a simple JavaScript object sanitization and validation module.

PatternPy

by keithorange

📈 PatternPy: A Python package revolutionizing trading analysis with high-speed pattern recognition, leveraging Pandas & Numpy. Effortlessly spot Head & Shoulders, Tops & Bottoms,…

amadeus

by constellation-rs

Harmonious distributed data analysis in Rust.