spark-sql

Make Your Company Data Driven. Connect to any data source, easily visualize, dashboard and share your data.

Better SQL in Jupyter. 📊

Practice Databricks coding skills with hands-on exercises. Import into Databricks Free Edition, write code, run assertions, check pass/fail. Covers Delta Lake, Spark SQL, PySpark, Auto Loader, medallion architecture, window functions, and more.

Apache Spark 3 - Structured Streaming Course Material

This repository will help you to learn about databricks concept with the help of examples. It will include all the important topics which we need in our real life experience as a data engineer. We will be using pyspark & sparksql for the development. At the end of the course we also cover few case studies.

Open-source graph query service layer for ClickHouse and Databricks, written in Rust, turning an existing database into a graph view in minutes. It also provides an embedded mode with chdb, now supporting writes.

Simplified ETL process in Hadoop using Apache Spark. Has complete ETL pipeline for datalake. SparkSession extensions, DataFrame validation, Column extensions, SQL functions, and DataFrame transformations

Sentiment Analysis of a Twitter Topic with Spark Structured Streaming

This project introduces PySpark, a powerful open-source framework for distributed data processing. We explore its architecture, components, and applications for real-time data analysis.

Sparglim✨ makes PySpark App Configurable and Deploy Spark Connect Server Easier!

A scalable data preprocessing framework built on PySpark for LLM training

A Big Data project leveraging AWS services and Apache frameworks to identify and visualize fraudulent credit card transaction patterns, providing actionable insights to mitigate financial fraud.

Demonstration of using Apache Spark to build robust ETL pipelines while taking advantage of open source, general purpose cluster computing.

A real time analytics dashboard to analyze the trending hashtags and @ mentions at any location using kafka and spark streaming.

A structured streaming was applied to the robot data from ROS-Gazebo simulation environment using Apache Spark. Data is collected in Kafka, analyzed by Apache Spark and stored in Cassandra.

This is a data processing pipeline that implements an End-to-End Real-Time Geospatial Analytics and Visualization multi-component full-stack solution, using Apache Spark Structured Streaming, Apache Kafka, MongoDB Change Streams, Node.js, React, Uber's Deck.gl and React-Vis, and using the Massachusetts Bay Transportation Authority's (MBTA) APIs …

ORM for Apache Spark and DataFrames schema manager

An naive anomaly detection and data visualization tool for F1 on board telemetry data.

Scalable ETL Pipeline: Processing 5M+ retail records with PySpark on GCP Dataproc. Automated the extraction of global business KPIs and consumer trends. Includes an Ethical Data Framework to ensure privacy and fairness at scale

This project builds a scalable log analytics pipeline use Lambda architecture for real-time and batch processing of NASA server logs.