YouTap
A data pipeline that collects YouTube's trending videos, analyses them and shows the results on a Kibana dashboard, all running in Docker containers.
- Period
- May 2023 – Jun 2023
- Status
- Completed
- Technologies
- 8
Overview
YouTap collects the most popular videos from the YouTube Data API and processes them in a streaming pipeline, from ingestion to visualisation.
Pipeline
- Collection — a Python script queries the YouTube API and publishes the data to Kafka.
- Processing — Logstash and Spark consume the stream and enrich the data.
- Storage and visualisation — Elasticsearch indexes the results, Kibana shows them on a dashboard.
Every service runs in Docker, orchestrated with Docker Compose; a Jupyter notebook walks through the project step by step.