Trender: GitHub Trending Analytics Platform

A batch analytics platform with a 3-layer data engineering pipeline (Raw → Staging → Analytics) that analyzes trending GitHub repositories across 3 programming languages (Python, TypeScript with Next.js >= 16, and Go). Leverages Render Workflows' distributed task execution to process data in parallel, storing results in a dimensional model for high-performance analytics.

Key Features

• Multi-Language Analysis: Tracks Python, TypeScript/Next.js, and Go repositories
• 3-Layer Data Pipeline: Raw ingestion → Staging validation → Analytics dimensional model
• Parallel Processing: 4 concurrent workflow tasks using Render Workflows SDK
• Render Ecosystem Spotlight: Dedicated showcase for Render-deployed projects
• Real-time Dashboard: Next.js 14 dashboard with analytics visualizations
• Hourly Updates: Automated cron job triggers workflow execution

Architecture

graph TD
    A[Cron Job Hourly] --> B[Workflow Orchestrator]
    B --> C[Python Analyzer]
    B --> D[TypeScript Analyzer]
    B --> E[Go Analyzer]
    B --> F[Render Ecosystem]
    C --> G[Raw Layer JSONB]
    D --> G
    E --> G
    F --> G
    G --> H[Staging Layer Validated]
    H --> I[Analytics Layer Fact/Dim]
    I --> J[Next.js Dashboard]

Loading

Tech Stack

Backend (Workflows)

• Python 3.11+
• Render Workflows SDK with @task decorators
• asyncpg for PostgreSQL
• aiohttp for async API calls
• GitHub REST API

Frontend (Dashboard)

• Next.js 14 (App Router)
• TypeScript
• Tailwind CSS
• Recharts for visualizations
• PostgreSQL (pg)

Infrastructure

• Render Workflows (task execution)
• Render Cron Job (hourly trigger)
• Render Web Service (Next.js dashboard)
• Render PostgreSQL (data storage)

Project Structure

trender/
├── workflows/
│   ├── workflow.py           # Main workflow with @task decorators
│   ├── github_api.py         # Async GitHub API client
│   ├── connections.py        # Shared resource management
│   ├── metrics.py            # Momentum/activity calculations
│   ├── render_detection.py  # Render usage detection
│   ├── etl/
│   │   ├── extract.py        # Raw layer extraction
│   │   ├── transform.py      # Staging transformations
│   │   ├── load.py           # Analytics layer loading
│   │   └── data_quality.py   # Quality scoring
│   └── requirements.txt
├── trigger/
│   ├── trigger.py            # Cron trigger script
│   └── requirements.txt
├── dashboard/
│   ├── app/                  # Next.js App Router pages
│   ├── lib/
│   │   └── db.ts            # Database utilities
│   └── package.json
├── database/
│   ├── schema/
│   │   ├── 01_raw_layer.sql
│   │   ├── 02_staging_layer.sql
│   │   ├── 03_analytics_layer.sql
│   │   └── 04_views.sql
│   └── init.sql
├── render.yaml
├── .env.example
└── README.md

Quick Start - Trigger a Workflow

If you've already completed the setup and just want to trigger a workflow run:

# Navigate to trigger directory
cd trigger

# Set environment variables
export RENDER_API_KEY=your_api_key
export RENDER_WORKFLOW_SLUG=trender-wf

# Install dependencies and run
pip install -r requirements.txt
python trigger.py

Or use the Render Dashboard: Workflows → trender-wf → Tasks → main_analysis_task → Run Task

---

Setup Instructions

Prerequisites

• GitHub authentication (Personal Access Token or OAuth App - covered in step 2)
• Render account
• Node.js 18+ (for dashboard)
• Python 3.11+ (for workflows)

1. Clone Repository

git clone <your-repo-url>
cd trender

2. GitHub Authentication Setup

Trender needs a GitHub access token to fetch repository data. You can choose between two authentication methods:

---

🎯 Option A: Personal Access Token (PAT) - Recommended

Best for: Individual developers, quick setup, local development

This is the simplest method - just create a token from GitHub settings.

Step 1: Run the auth setup script

cd workflows
pip install -r requirements.txt
python auth_setup.py

Step 2: Choose option [1] for PAT

Step 3: Follow the interactive prompts:

1. Open https://github.com/settings/tokens/new in your browser
2. Configure the token:
   • Note: Trender Analytics Access
   • Expiration: No expiration (or your preference)
   • Scopes:
     • ✓ repo (Full control of private repositories)
     • ✓ read:org (Read org and team membership)
3. Click "Generate token"
4. Copy the token (starts with ghp_ or github_pat_)
5. Paste it into the terminal when prompted

Step 4: Save your token

The script will verify your token and display:

✅ SUCCESS! Your GitHub access token (PAT):
============================================================

ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

============================================================
Add this to your .env file:
GITHUB_ACCESS_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Add the token to your .env file:

GITHUB_ACCESS_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

✅ Done! Skip to Step 3.

---

🔐 Option B: OAuth App - Advanced

Best for: Team setups, production deployments, requiring user authorization flow

Step 1: Create GitHub OAuth App

1. Go to https://github.com/settings/developers
2. Click "New OAuth App"
3. Fill in the details:
   • Application name: Trender Analytics
   • Homepage URL: http://localhost:3000
   • Authorization callback URL: http://localhost:8000/callback
4. Click "Register application"
5. Note your Client ID (starts with Ov23 or Iv1.)
6. Click "Generate a new client secret" and save it

Step 2: Add OAuth credentials to .env

GITHUB_CLIENT_ID=Ov23xxxxx_or_Iv1.xxxxx
GITHUB_CLIENT_SECRET=your_secret_here

Step 3: Run the auth setup script

cd workflows
pip install -r requirements.txt
python auth_setup.py

Choose option [2] for OAuth, then:

1. The script starts a local server on port 8000
2. Your browser opens to GitHub's authorization page
3. Click "Authorize" to approve
4. The script exchanges the auth code for a token
5. Your GITHUB_ACCESS_TOKEN is displayed

Step 4: Save your token

Add the token to your .env file:

GITHUB_ACCESS_TOKEN=gho_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

---

🔒 Security Best Practices (Both Methods)

• ✅ Tokens don't expire (unless you set expiration on PAT)
• ✅ Never commit tokens to version control (.env is in .gitignore)
• ✅ Token scopes: repo and read:org only
• ✅ Revoke access anytime at https://github.com/settings/tokens
• ⚠️ Treat tokens like passwords

🐛 Troubleshooting

PAT Issues:

• Token doesn't start with ghp_: Classic tokens start with ghp_, fine-grained tokens with github_pat_
• API returns 401: Token may be expired or revoked. Generate a new one.
• Rate limit errors: Ensure token has proper scopes selected

OAuth Issues:

• Port 8000 in use: Run lsof -ti:8000 | xargs kill -9, then try again
• "Redirect URI mismatch": Ensure callback URL in OAuth app is exactly http://localhost:8000/callback
• Browser doesn't open: Manually visit the URL shown in the terminal
• "Bad verification code": Code expires quickly. Run python auth_setup.py again

Both Methods:

• Token verification fails: Check your internet connection
• Need to regenerate: Revoke old token at https://github.com/settings/tokens and generate new one

3. Set Up Environment Variables

cp .env.example .env
# Edit .env with your credentials

Your .env file should now contain (from step 2):

If you used PAT (Option A):

GITHUB_ACCESS_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

If you used OAuth (Option B):

GITHUB_CLIENT_ID=Ov23xxxxx_or_Iv1.xxxxx
GITHUB_CLIENT_SECRET=your_secret_here
GITHUB_ACCESS_TOKEN=gho_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Other required variables (add as you complete the setup):

• DATABASE_URL: PostgreSQL connection string (from step 4)
• RENDER_API_KEY: Render API key (from https://dashboard.render.com/u/settings#api-keys)
• RENDER_WORKFLOW_SLUG: trender-wf (or your workflow slug from step 6)

4. Create PostgreSQL Database on Render

1. Go to Render Dashboard
2. Create new PostgreSQL database named trender
3. Note the connection string for DATABASE_URL

5. Initialize Database Schema

Option 1: Using the init.sql script (Recommended)

# Connect to your Render PostgreSQL instance and run the initialization script

DATABASE_URL=YOUR_DATABASE_URL
psql $DATABASE_URL -f database/init.sql

Option 2: Run schema files individually

If you prefer to run the schema files one at a time:

# Run each schema file in order
psql $DATABASE_URL -f database/schema/01_raw_layer.sql
psql $DATABASE_URL -f database/schema/02_staging_layer.sql
psql $DATABASE_URL -f database/schema/03_analytics_layer.sql
psql $DATABASE_URL -f database/schema/04_views.sql

What gets created:

Raw Layer:

• raw_github_repos: Stores complete GitHub API responses
• raw_repo_metrics: Stores repository metrics (stars, forks, issues)

Staging Layer:

• stg_repos_validated: Cleaned and validated repository data
• stg_render_enrichment: Render-specific metadata and detection

Analytics Layer:

• Dimension tables: dim_repositories, dim_languages, dim_render_services
• Fact tables: fact_repo_snapshots, fact_render_usage, fact_workflow_executions

Views:

• Pre-aggregated analytics views for dashboard queries

Verify Database Initialization

Check that all tables were created successfully:

psql $DATABASE_URL -c "\dt"

You should see 12+ tables across the raw, stg, dim, and fact prefixes.

Troubleshooting

• Connection refused: Ensure your DATABASE_URL is correct and the Render PostgreSQL instance is active
• Permission denied: Make sure you're using the connection string with full admin privileges
• Tables already exist: Drop the database and recreate it, or use DROP TABLE IF EXISTS statements

6. Deploy Services via render.yaml

The render.yaml file defines:

• Web Service: Next.js dashboard
• Cron Job: Hourly workflow trigger
• Database: PostgreSQL instance

Deploy to Render:

# Push to GitHub and connect to Render
# Or use Render Blueprint button

7. Configure Environment Variables in Render

After deploying via render.yaml, add your GitHub access token to the workflow service (trender-wf) in the Render Dashboard:

1. Go to your trender-wf workflow in Render Dashboard
2. Navigate to Environment tab
3. Add:
   • GITHUB_ACCESS_TOKEN: The token you generated in step 2 (starts with ghp_ or gho_ or github_pat_)
   • DATABASE_URL: Automatically connected from the database (no action needed)

Important: After adding the token, trigger a manual deploy:

• Click "Manual Deploy" → "Clear build cache & deploy"
• This ensures the environment variables are available to your workflow tasks

Note: You only need GITHUB_ACCESS_TOKEN in Render. If you used OAuth, you don't need to add GITHUB_CLIENT_ID or GITHUB_CLIENT_SECRET to Render.

8. Trigger Workflow Runs

There are three ways to trigger a workflow run to populate data:

Method 1: Using the Trigger Script (Recommended)

The trigger/trigger.py script uses the Render SDK to trigger workflows programmatically:

cd trigger

# Install dependencies
pip install -r requirements.txt

# Set required environment variables
export RENDER_API_KEY=your_render_api_key
export RENDER_WORKFLOW_SLUG=trender  # Your workflow slug from Render dashboard

# Run the trigger script
python trigger.py

Expected output:

Triggering task: trender/main-analysis-task
✓ Workflow triggered successfully at 2026-01-20 12:00:00
  Task Run ID: run_abc123xyz
  Initial Status: running

Method 2: Using the Render Dashboard

1. Go to Render Dashboard
2. Navigate to Workflows section
3. Select your trender workflow
4. Click "Trigger Workflow" button
5. Select the main-analysis-task task
6. Click "Run Task"

Method 3: Using Render CLI

If you have the Render CLI installed:

# Install Render CLI (if not already installed)
npm install -g @render-inc/cli

# Login to Render
render login

# Trigger the workflow
render workflows trigger trender main-analysis-task

Verify Workflow Execution

Check the workflow status:

1. Via Dashboard: Go to Workflows → trender → View recent runs
2. Via Script: The trigger script outputs the Task Run ID
3. Via Database: Query the fact_workflow_executions table:

psql $DATABASE_URL -c "SELECT * FROM fact_workflow_executions ORDER BY execution_date DESC LIMIT 1;"

Expected workflow completion time: 8-15 seconds for ~300 repositories

Troubleshooting

• "RENDER_API_KEY not set": Export your API key from Render Settings
• "Task not found": Verify your workflow slug and that the workflow is deployed
• "Connection refused": Check that DATABASE_URL is correct and the database is running
• Workflow fails: Check the Render dashboard logs for detailed error messages

9. Access Dashboard

Once the workflow completes, access your dashboard at:

https://trender-dashboard.onrender.com

You should see:

• Top trending repositories across Python, TypeScript, and Go
• Render ecosystem projects
• Momentum scores and analytics
• Historical trends

Data Pipeline Layers

Layer 1: Raw Ingestion

• Stores complete GitHub API responses
• Tables: raw_github_repos, raw_repo_metrics
• Purpose: Audit trail and reprocessing capability

Layer 2: Staging (Validation)

• Cleaned and validated data
• Tables: stg_repos_validated, stg_render_enrichment
• Data quality scoring (0.0 - 1.0)
• Business rules applied

Layer 3: Analytics (Dimensional Model)

• Dimensions: dim_repositories, dim_languages, dim_render_services
• Facts: fact_repo_snapshots, fact_render_usage, fact_workflow_executions
• Views: Pre-aggregated analytics for dashboard

Workflow Tasks

The workflow consists of 8 tasks decorated with @task:

1. main_analysis_task: Orchestrator that spawns parallel tasks
2. fetch_language_repos: Fetches repos for Python, TypeScript, Go
3. analyze_repo_batch: Analyzes repos in batches of 10
4. fetch_render_ecosystem: Fetches Render-related projects
5. analyze_render_projects: Analyzes Render-specific features
6. aggregate_results: ETL pipeline execution (Extract → Transform → Load)
7. store_execution_stats: Records workflow performance metrics

Metrics Calculated

• Star Velocity: (stars_last_7_days / total_stars) * 100
• Activity Score: Weighted formula using commits, issues, contributors
• Momentum Score: (star_velocity * 0.4) + (activity_score * 0.6)
• Render Boost: 1.2x multiplier for projects using Render
• Freshness Penalty: 0.9x for repos older than 180 days

Development

Local Development - Workflows

cd workflows
pip install -r requirements.txt
python workflow.py

Local Development - Dashboard

cd dashboard
npm install
npm run dev
# Access at http://localhost:3000

Run Database Migrations

psql $DATABASE_URL -f database/schema/01_raw_layer.sql
psql $DATABASE_URL -f database/schema/02_staging_layer.sql
psql $DATABASE_URL -f database/schema/03_analytics_layer.sql
psql $DATABASE_URL -f database/schema/04_views.sql

Success Metrics

Technical:

• Process 300+ repos across 3 languages in under 10 seconds
• 3x speedup vs sequential processing
• 99%+ success rate on workflow runs
• Data quality score >= 0.90 for 95%+ repositories

Marketing:

• Showcase 50+ Render ecosystem projects
• Track Render adoption vs competitors
• Identify case study candidates

License

MIT

Contributing

Contributions welcome! Please open an issue or submit a pull request.