Developer Ecosystems — Code Hosts, Package Registries, Dev Community, CI

50+ sources. Field ratings 1-5 (5 = best). Current as of May 2026.

1. Code Hosting Platforms

GitHub (REST + GraphQL + Webhooks)

• REST: api.github.com / GraphQL: api.github.com/graphql / Webhooks: ~70 event types real-time
• Auth: PAT, GitHub App (recommended for prod), OAuth, anonymous (limited)
• Pricing: Free for public data. Enterprise Cloud for org-level scale
• Rate limits: 60/hr anonymous; 5,000/hr authenticated user; 15,000/hr GitHub Apps on Enterprise Cloud; 10/min secondary search limit; GraphQL 5,000 points/hr
• Data richness: 5 | Stability: 2 | Legal: Public data redistribution permitted with attribution caveats; AUP prohibits training general LLMs on private data | DX: 5 | Viability: 5
• Gotchas: Secondary rate limits vague + bursty; pagination caps at 1,000 results for most search endpoints; archived repos drop from search

GH Archive (gharchive.org)

• URL: https://www.gharchive.org / BigQuery: githubarchive.day.*
• Exposes: Every public GitHub event since 2011-02, ~3-5M events/day
• Auth: None (HTTP) / GCP for BigQuery
• Pricing: Free dumps / BigQuery on-demand ($6.25/TB)
• Data richness: 5 (historical depth nothing else matches) | Stability: 2 (community-run by Ilya Grigorik, donations-funded; SPOF) | DX: 4 | Viability: 5
• Gotchas: Event schema changed in 2015; ~50GB/month compressed

Other code hosts

• GitLab API (SaaS + self-hosted) — 4/5 viability; 2,000 req/min authenticated on gitlab.com
• Bitbucket Cloud — 2/5 viability (Atlassian deprecated issues for new repos 2023; app passwords sunset Sept 2025)
• SourceForge — 2/5 viability (legacy archaeology only)
• Codeberg (Gitea-based, non-profit, EU) — 3/5 viability; growing in FOSS/EU
• Gitea / Forgejo (self-hosted) — 3/5 viability; APIs diverging post-2023 relicense
• Sourcehut (sr.ht) — 2/5 viability; explicit anti-ML-training ToS — respect it

2. Package Registries

npm Registry

• URL: https://registry.npmjs.org / https://api.npmjs.org (downloads)
• Auth: None for read; token for publish
• Pricing: Free; npm Pro/Teams/Enterprise for private
• Rate limits: Soft, CDN-fronted; replication via CouchDB _changes feed
• DR: 5 | Stability: 2 | DX: 5 | Viability: 5
• Gotchas: No version-level download stats; replication endpoint is the bulk path (replicate.npmjs.com/_changes)

PyPI (Warehouse)

• URL: https://pypi.org / JSON at pypi.org/pypi/<pkg>/json
• PyPI Downloads via BigQuery: bigquery-public-data.pypi.file_downloads — every PyPI download with timestamp, country, installer, Python version, OS
• DR: 5 (richest install telemetry of any registry) | Stability: 1 (PSF/Fastly/AWS) | DX: 5 | Viability: 5
• Gotchas: XML-RPC endpoints deprecated 2024-2025 → migrate to JSON + RSS + Simple API. Costly BigQuery queries — always filter by _TABLE_SUFFIX

Other registries

• RubyGems — 5 DR (version-level downloads — better than npm); Ruby Central foundation backed
• Crates.io — 5 DR; Rust Foundation backed; User-Agent w/ contact email required (403 without)
• Maven Central — 3 DR (no download counts publicly!); Sonatype acquired by EQT 2023; new central.sonatype.com portal 2024
• NuGet — 5 DR (catalog gives full event log); Microsoft-hosted; very stable
• Go Module Proxy / Index — 3 DR (no downloads); Google-hosted; pkg.go.dev richer metadata but no docs
• Packagist (PHP) — 4 DR; v2 metadata format reduced bandwidth
• Homebrew (formulae.brew.sh) — 4 DR; 30d/90d/365d install analytics; opt-out rate ~10-15%
• Docker Hub — 3 DR; anonymous 100 pulls/6hr per IP (pricing churn 2020+)
• GitHub Container Registry — 3 DR; pull stats not exposed publicly
• Artifact Hub (CNCF) — 4 DR; Helm + OPA + Falco + Krew + Tekton + Argo + KEDA in one API
• JSR (Deno) — 4 DR; TypeScript-first, launched March 2024
• Conda / Conda-Forge / Anaconda.org — 4 DR; Anaconda Inc. 2024 paywall for ≥200-employee orgs
• Hex (Elixir), CRAN (R) — niche but stable
• CocoaPods — maintenance mode since 2023; SPM (Swift Package Index) is the future

3. Package Metadata & Insight Aggregators

4. Code Search & Archive

• Sourcegraph — 1/5 viability for Vivly: hosted cloud terminated Feb 2024; self-host only
• Software Heritage Archive — 5/5 viability; UNESCO + Inria backed; ~20B unique files, ~3B commits archived
• Common Crawl — 3/5 for code-specific (not native, but useful for doc cross-ref)
• GH Trending — 3/5; no official API, scrape HTML
• papers-with-code — 3/5; Meta-owned, minimal staffing 2024-2025; still useful for ML repo discovery
• Hugging Face Hub — 5/5 viability; THE ML ecosystem source

5. Developer Communities

• DEV.to (Forem) — 4/5; CC-licensed content; 30 req/30sec write; reads liberal
• Hashnode — 3/5; GraphQL only (schema churned during 2023-2024 headless pivot)
• Stack Overflow / Stack Exchange API — 3/5 viability (traffic declining ~30% YoY post-ChatGPT, but data still rich). 300/day anonymous, 10K/day with key. 2024 ToS update claims ML training rights; CC-BY-SA attribution required for redistribution
• Stack Exchange Data Dump — 4/5; quarterly XML on archive.org (~100GB compressed)
• Lobste.rs — 3/5; small but elite tech audience; invite-only
• Hacker News / Show HN — 5/5; coordinate with social agent

6. CI / Build

• GitHub Actions API — 5/5 viability; logs auto-delete after 90 days
• CircleCI API — 3/5 viability (2023 security incident, layoffs 2024)
• Travis CI — 1/5 (essentially abandoned)
• Buildkite — 3/5; Australian indie; well-run
• Jenkins — 2/5 (mostly behind firewalls; can't aggregate)
• Drone CI — 2/5 (Harness-owned, eclipsed)
• Woodpecker CI (Drone fork) — 2/5

7. Specialized / Notable

• CNCF DevStats — 5 DR for CNCF orbit (Kubernetes, etc.); Postgres dump available
• OSS Insight (PingCAP) — 4 DR; vendor demo for TiDB; PingCAP layoffs 2024
• GitHub Sponsors API — 4 viability (funding flow signal)
• Open Collective — 3 viability (2024 governance disruption; platform continues but legal uncertainty)
• Cauldron.io — 2 viability (CHAOSS-based; academic-led)
• Open Hub (Ohloh) — 1 viability (frozen since 2018)

Vivly-Specific Recommendations

Tier 1 — must-have, build first (high data richness, low stability risk, permissive legal):

• GH Archive (BigQuery), GitHub REST + GraphQL + Webhooks, npm Registry + replication, PyPI + BigQuery downloads, deps.dev (BigQuery + API), OSV, OpenSSF Scorecard, ecosyste.ms suite, crates.io, RubyGems, NuGet, Maven Central, Go proxy, Packagist, Software Heritage, Stack Exchange API + Data Dump, Hacker News.

Tier 2 — high value, watch stability:

• GitLab API, DEV.to + Hashnode, Homebrew analytics, Artifact Hub, JSR, Hugging Face, GitHub Sponsors + Open Collective, OSS Insight + CNCF DevStats (dashboards as reference).

Tier 3 — niche/conditional:

• Bitbucket (declining), SourceForge (legacy archaeology), Codeberg/Forgejo (small but ethically aligned), CircleCI/Buildkite (CI minority share), Bundlephobia/Packagephobia (self-host), papers-with-code (irreplaceable for ML, unmaintained), GH Trending scrape.

Avoid / off-limits:

• sr.ht — explicit anti-ML-training ToS
• Replit — no supported public API
• libraries.io — stale; use ecosyste.ms
• Snyk Advisor / Tidelift — ToS-restricted commercial
• Sourcegraph hosted — gone
• Open Hub — frozen

Architectural notes:

1. GH Archive + BigQuery deps.dev + Software Heritage form a "canonical triangle" — events, package-graph, archived source. Build everything else atop these.
2. Treat aggregators (libraries.io, ecosyste.ms, deps.dev) as cross-references, not sources of record. Always link back to primary registry/Git remote.

Legal hygiene:

• CC-BY-SA attribution chain for Stack Exchange content
• Honor robots.txt and ToS (GH Trending scrape, GitLab self-hosted)
• Never bulk-scrape Sourcehut
• Track Stack Overflow's 2024 ToS — derived datasets may trigger attribution obligations

2024-2026 Changes Watch