Top Data & Analytics tools for data engineering โ made for analysts, businesses, researchers and data-curious users. Every project below is a real, public GitHub repository, ranked by the stars the community gave it.
50 repositories ยท ranked by GitHub stars by default
๐ท๏ธ The BEST Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here forโฆ
The 30 Days of Python programming challenge is a step-by-step guide to learn the Python programming language in 30 days. This challenge may take more than 100 days. Follow yourโฆ
Apache Superset is a Data Visualization and Data Exploration Platform
LlamaIndex is the document processing platform for AI
๐ค Powerful asynchronous state management, server-state utilities and data fetching for the web. TS/JS, React Query, Solid Query, Svelte Query and Vue Query.
The easy-to-use open source Business Intelligence and Embedded Analytics tool that lets everyone work with data :bar_chart:
Learn how to develop, deploy and iterate on production-grade ML applications.
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
Data Engineering Zoomcamp is a free 9-week course on building production-ready data pipelines. Join the course here ๐๐ผ
This is a repo with links to everything you'd ever want to learn about data engineering
๐ SheetJS Spreadsheet Data Toolkit -- New home https://git.sheetjs.com/SheetJS/sheetjs
React Hooks for Data Fetching
๐ Papers & tech blogs by companies sharing their work on data science & machine learning in production.
Orchestrate everything, everywhere. The open-source control plane for workflows across data, infrastructure, business processes and AI agents. Declarative YAML, event-driven, anyโฆ
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
Chat with your database or your datalake (SQL, CSV, parquet). PandasAI makes data analysis conversational using LLMs and RAG.
AKShare is an elegant and simple financial data interface library for Python, built for human beings! ๅผๆบ่ดข็ปๆฐๆฎๆฅๅฃๅบ
Open-source data movement for ELT pipelines and AI agents โ from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.
Turns Data and AI algorithms into production-ready web applications in no time.
Data and code behind the articles and graphics at FiveThirtyEight
Workflow Engine for Kubernetes
The official home of the Presto distributed SQL query engine for big data
An orchestration platform for the development, production, and observation of data assets.
Generate massive amounts of fake data in the browser and node.js
The Data Engineering Cookbook
A curated list of awesome big data frameworks, ressources and other awesomeness.
Graph-Native Infrastructure for Context and Accountable AI Systems
Roadmap to becoming a data engineer in 2021
Always know what to expect from your data.
Incremental engine for long horizon agents ๐ Star if you like it!
:orange_book: ไธญๅๆฐๅๅญๅ ธๆฐๆฎๅบใๅ ๆฌๆญๅ่ฏญ๏ผๆ่ฏญ๏ผ่ฏ่ฏญ๏ผๆฑๅญใ
A configuration as code language with rich validation and tooling.
PRQL is a modern language for transforming data โ a simple, powerful, pipelined SQL replacement
The agent platform built on sync.
:card_index: A simple fake data generator for C#, F#, and VB.NET. Based on and ported from the famed faker.js.
๐ Python-powered shell. Full-featured, cross-platform and AI-friendly.
Event streaming platform for agentic AI. Continuously ingest, transform, and serve event streams in real time, at scale.
A web interface to create custom vector-based visualizations on top of RAWGraphs core
๐ง Build, run, and manage data pipelines for integrating and transforming data.
Fancy stream processing made operationally mundane
Open Source Feature Flags, Experimentation, and Product Analytics
[SIGMOD'27] Easy Data Preparation with latest LLMs-based Operators and Pipelines.
A roadmap connecting many of the most important concepts in machine learning, how to learn them and what tools to use to perform them.
Interactive Tables and Data Grids for JavaScript
LLM-Driven Extraction of Unstructured Data โ Built for API Deployments & ETL Pipeline Workflows
The Open Source Feature Store for AI/ML
Data processing for and with foundation models! ๐ ๐ ๐ฝ โก๏ธ โก๏ธ๐ธ ๐น ๐ท
Business intelligence as code: build fast, interactive data visualizations in SQL and markdown
The leader in Customer Data Infrastructure
2027 AI/ML internship & new graduate job list updated daily