Add QCEW CSV module, ECEC breakdown, response caching + typed errors
Item 1 — QCEW county/industry detail: - new bls_client/qcew.py: thin client over the QCEW Open Data CSV service (area/industry/size slices; no API key, no quota), with filter_rows helper - covers the county/industry detail the timeseries API can't reach Item 2 — ECEC benefit breakdown: - new series.ecec() builder from the authoritative cm.estimate decode table - real helpers: wages, total benefits, paid leave, supplemental pay, health insurance, retirement & savings, legally required + ecec_dashboard() - FIX: ecec_total_benefits was CMU1036... (education/health industries only); correct all-civilian total benefits is CMU1030000000000D Item 3 — caching + typed errors: - bls_client/cache.py FileCache (on-disk, TTL); BLSClient(cache=True), queries_used counter (cache hits don't spend quota) - bls_client/errors.py: BLSError / BLSQuotaError / BLSRequestError; quota exhaustion now raises a clear BLSQuotaError instead of a generic RuntimeError Tests: 119 offline (+24) incl. ECEC goldens, QCEW fixture parse, cache/quota behavior; live smoke extended to ECEC + QCEW. Helper sweep: 96/96 live. README: caching/QCEW usage, updated coverage + limitations. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
74
bls_client/qcew.py
Normal file
74
bls_client/qcew.py
Normal file
@ -0,0 +1,74 @@
|
||||
"""QCEW Open Data Access — county/industry employment & wages.
|
||||
|
||||
The BLS *timeseries* API only carries QCEW national/state totals. The full
|
||||
QCEW detail (every county, every NAICS industry, establishment counts and
|
||||
wages) lives behind a separate CSV service that needs no API key and counts
|
||||
against no quota:
|
||||
|
||||
https://data.bls.gov/cew/data/api/{year}/{qtr}/{slice}/{code}.csv
|
||||
|
||||
This module is a thin wrapper over that service. It is deliberately separate
|
||||
from BLSClient (different API, CSV not JSON, no key). Each call returns a list
|
||||
of plain dict rows (the CSV columns), so it composes with csv/pandas directly.
|
||||
|
||||
Quarter argument: 1-4 for a specific quarter, or "a" for annual averages.
|
||||
Area FIPS examples: "US000" national, "11000" DC (state), "11001" a county,
|
||||
metro codes start with "C". Industry codes are NAICS (e.g. "10" = all
|
||||
industries, "722" = food services, "622" = hospitals).
|
||||
"""
|
||||
import csv
|
||||
import io
|
||||
|
||||
import requests
|
||||
|
||||
QCEW_BASE = "https://data.bls.gov/cew/data/api"
|
||||
_HEADERS = {"User-Agent": "bls-data-library (https://gitea.doesworks.net/giteaadmin/bls-data)"}
|
||||
|
||||
# A few common columns, for reference / convenience:
|
||||
# area_fips, own_code, industry_code, agglvl_code, size_code, year, qtr,
|
||||
# annual_avg_estabs, annual_avg_emplvl, total_annual_wages, annual_avg_wkly_wage,
|
||||
# avg_annual_pay (quarterly slices use month1_emplvl/qtrly_estabs/total_qtrly_wages, etc.)
|
||||
|
||||
OWNERSHIP = {
|
||||
"total": "0", "federal": "1", "state": "2", "local": "3",
|
||||
"private": "5", "government": "8",
|
||||
}
|
||||
|
||||
|
||||
def _fetch_csv(url: str) -> list[dict]:
|
||||
r = requests.get(url, timeout=60, headers=_HEADERS)
|
||||
r.raise_for_status()
|
||||
return parse_csv(r.text)
|
||||
|
||||
|
||||
def parse_csv(text: str) -> list[dict]:
|
||||
"""Parse QCEW CSV text into a list of row dicts (BLS quotes every field)."""
|
||||
return list(csv.DictReader(io.StringIO(text)))
|
||||
|
||||
|
||||
def area(area_fips: str, year: int, qtr="a") -> list[dict]:
|
||||
"""Every industry × ownership for one area (county/state/metro/national)."""
|
||||
return _fetch_csv(f"{QCEW_BASE}/{year}/{qtr}/area/{area_fips}.csv")
|
||||
|
||||
|
||||
def industry(industry_code: str, year: int, qtr="a") -> list[dict]:
|
||||
"""One NAICS industry across all areas."""
|
||||
return _fetch_csv(f"{QCEW_BASE}/{year}/{qtr}/industry/{industry_code}.csv")
|
||||
|
||||
|
||||
def size(size_code: str, year: int) -> list[dict]:
|
||||
"""Establishment-size breakdown (size data is published for Q1 only)."""
|
||||
return _fetch_csv(f"{QCEW_BASE}/{year}/1/size/{size_code}.csv")
|
||||
|
||||
|
||||
def filter_rows(rows, own_code=None, industry_code=None, agglvl_code=None) -> list[dict]:
|
||||
"""Filter QCEW rows by ownership / industry / aggregation-level code."""
|
||||
out = rows
|
||||
if own_code is not None:
|
||||
own = OWNERSHIP.get(own_code, own_code)
|
||||
out = [r for r in out if r.get("own_code") == own]
|
||||
if industry_code is not None:
|
||||
out = [r for r in out if r.get("industry_code") == industry_code]
|
||||
if agglvl_code is not None:
|
||||
out = [r for r in out if r.get("agglvl_code") == agglvl_code]
|
||||
return out
|
||||
Reference in New Issue
Block a user