DOES-Applications/bls-data
5
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add project overview & usage guide (docs/BLS_Library_Overview.md)

Browse Source 
Narrative of the audit/repair work plus a practical usage guide for the
client, query helpers, series builders, QCEW module, caching/errors/retry,
testing, and coverage.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
David Boyd
2026-06-22 10:51:57 -04:00
parent 0d0d6dabe2
commit 8f0122011a
1 changed files with 293 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

293
docs/BLS_Library_Overview.md Normal file
View File

@ -0,0 +1,293 @@
# BLS Data Library — Overview & Usage Guide
**Repository:** `giteaadmin/bls-data` on `gitea.doesworks.net`
**Status:** feature-complete · 96/96 query helpers verified live · 131 tests · pip-installable
**Last updated:** June 22, 2026
This document covers two things: **what the library is and how it got here**, and a
**practical guide to using it**. For the terse reference, see `README.md`,
`USAGE.md`, and `series_id_formats.md`.
---
## 1. What this is
A dependency-light Python client for the U.S. Bureau of Labor Statistics (BLS) API.
Its job is to make BLS data retrievable **without hand-encoding series IDs** — the
cryptic 11-to-25-character codes BLS uses as primary keys (e.g.
`OEUN000000000000015125213` for "national median annual wage, software developers").
The library wraps two BLS data services:
| Service | What it serves | Key? | Quota? |
|---|---|---|---|
| **Timeseries API** (`api.bls.gov/publicAPI/v2`) | Most surveys — unemployment, payrolls, prices, wages, openings, productivity | yes (free) | 500/day |
| **QCEW Open Data** (`data.bls.gov/cew/data/api`) | County- and industry-level employment & wages | no | none |
It provides: a batching/caching HTTP client, ~96 pre-built "query helpers" for common
series, low-level series-ID builders for everything else, typed errors, retry/backoff,
and a test suite that locks the encodings in place.
---
## 2. What we did (engineering summary)
The library existed but was **substantially broken** — a full audit found only 58 of 82
query helpers returned live data. Whole survey modules emitted series IDs that BLS
rejected. The work fell into four phases:
### Phase 1 — Fix the broken encodings
Every dead helper traced back to a wrong series-ID encoding (mostly wrong field widths
inherited from an internally-inconsistent reference doc). Fixed and verified against the
live API:
| Survey | Bug | Fix |
|---|---|---|
| **JOLTS** | IDs were 18 chars (12 zeros) | correct format is 21 chars / 15 zeros (added state+area+sizeclass fields) |
| **OES** | national area code hard-coded `0000400` | corrected to `0000000` |
| **ECI** | wages/benefits put as the trailing char | trailing char is the *estimate* code; component is a mid-string field; published unadjusted (CIU) |
| **Productivity** | 2-digit sector + 3-digit measure | real format is 4-digit each (e.g. `PRS85006092`) |
| **PPI** | "finished goods" index (discontinued by BLS) | repointed to **final demand** (`WPUFD4`) |
| **SOC code** | software developers `151132` (2010 SOC, retired) | `151252` (2018 SOC) |
Result: **82/82 helpers live.**
### Phase 2 — Make it shareable
- API key moved to the `BLS_API_KEY` environment variable (config file gitignored; the
key was never in git history).
- Added `requirements.txt`; rewrote `README.md` as a proper project front door.
### Phase 3 — Package + test
- `pyproject.toml` — the library is now `pip install`-able; builds a wheel and sdist.
- A **pytest suite** that locks every series-ID encoding against known-good IDs (offline,
no quota) plus a live smoke test behind an opt-in `-m live` marker. This is the
regression guard — it fails instantly if an encoding is ever re-broken.
### Phase 4 — Complete the coverage
Three gaps closed in one efficient pass (total cost: ~2 of the 500 daily API queries,
because the decode tables and QCEW data are off-quota and ECEC verification batched into
a single call):
1. **QCEW county/industry detail** — a new `qcew` module over the separate CSV service,
reaching the per-county, per-industry data the timeseries API can't.
2. **ECEC benefit breakdown** — health insurance, retirement, paid leave, etc., built
from BLS's authoritative decode table (not guessed). This also caught a latent bug:
`ecec_total_benefits` had pointed at an education/health-industry series rather than
the all-civilian total.
3. **Caching + typed errors** — an on-disk response cache (so reruns don't re-spend
quota) and a `BLSQuotaError` that distinguishes "you're throttled" from "your series
ID is wrong" — the exact ambiguity that had previously stalled debugging.
Plus a **retry/backoff layer** for transient network failures (timeouts, `429`/`5xx`),
which never retries quota rejections or 4xx.
**End state:** 96/96 helpers live, 131 tests, every survey at full useful depth.
---
## 3. Architecture
```
bls_client/
├── client.py BLSClient — batching, caching, retry, named fetches, row flattening
├── series.py low-level series-ID builders (every survey)
├── qcew.py QCEW Open Data CSV client (county/industry; no key, no quota)
├── cache.py on-disk response cache (FileCache)
├── retry.py exponential-backoff retry for transient network errors
├── errors.py BLSError / BLSQuotaError / BLSRequestError
└── queries/
├── employment.py payrolls, unemployment (LAUS), JOLTS, QCEW totals
├── prices.py CPI, PPI, average prices, import/export prices
├── wages.py OES occupational wages, ECI, ECEC breakdown
└── productivity.py major-sector productivity & costs
```
Two layers: **query helpers** (`queries/…`) return ready-to-use series IDs for common
asks; **series builders** (`series.py`) construct any valid ID from parameters when a
helper doesn't exist.
---
## 4. Install & configure
```bash
pip install -r requirements.txt # just `requests`
# or, to install the package itself:
pip install -e .
```
Get a free API key (instant): https://data.bls.gov/registrationEngine/ — 500 queries/day.
```bash
cp config.example.py config.py
export BLS_API_KEY="your-key" # config.py reads this env var; takes precedence
```
`config.py` is gitignored. You can also pass the key directly: `BLSClient("your-key")`.
> **Note:** on a PEP-668 "externally managed" Python (e.g. recent Ubuntu), install into a
> virtualenv: `python3 -m venv .venv && . .venv/bin/activate && pip install -e ".[dev]"`.
---
## 5. Usage
### The client
```python
from bls_client import BLSClient
client = BLSClient(API_KEY, cache=True) # caching on (recommended)
# Fetch by series ID(s) — auto-batches >50 series
data = client.fetch(["CES0000000001"], 2020, 2025)
# Most recent N years
data = client.fetch_latest("CES0000000001", years=1)
# Labeled dict in, labeled dict out
data = client.fetch_named({"Payrolls": "CES0000000001"}, 2024, 2025)
# Helpers for reading results
obs = BLSClient.latest_obs(next(iter(data.values()))) # most recent observation
rows = BLSClient.to_rows(data) # flat rows for CSV/pandas
```
### Query helpers (the common case)
```python
from bls_client.queries import employment, prices, wages, productivity
employment.nonfarm_payrolls() # headline jobs number
employment.dc_region_unemployment() # DC/MD/VA + 3 metros (labeled dict)
employment.jolts_dashboard() # openings/hires/quits/layoffs/separations
prices.cpi_dashboard() # CPI: all-items, core, food, energy, …
prices.ppi_final_demand()
wages.occupation_annual_median_wage("151252") # any SOC code (software devs here)
wages.ecec_dashboard() # full employer cost-per-hour breakdown
wages.eci_total_compensation()
productivity.productivity_dashboard() # output/hr, ULC, etc.
```
Putting it together:
```python
client = BLSClient(API_KEY, cache=True)
result = client.fetch_named(wages.ecec_dashboard(), 2024, 2024)
for label, series in result.items():
obs = BLSClient.latest_obs(series)
print(f"{label:22} ${obs['value']}/hr")
# Total Compensation $47.20/hr
# Health Insurance $3.54/hr
# Retirement & Savings $2.45/hr ...
```
### Series builders (anything not pre-built)
```python
from bls_client import series
series.laus_state(48, "rate") # Texas unemployment rate
series.ces_national("manufacturing") # manufacturing payrolls
series.cpi("gasoline", seasonal=True)
series.oes_national("291141", data_type="annual_median") # registered nurses
series.ecec("health_insurance", owner="private")
```
### QCEW — county & industry detail
The timeseries helpers give QCEW national/state totals; the `qcew` module reaches the
full detail via the separate CSV service (no key, no quota):
```python
from bls_client import qcew
rows = qcew.area("11000", 2024, "a") # everything for DC, annual averages
dc_private = qcew.filter_rows(rows, own_code="private", industry_code="10", agglvl_code="51")
# -> establishments, employment, total wages, avg weekly wage for DC private sector
hospitals = qcew.industry("622", 2024, "a") # one NAICS industry across all areas
```
Each call returns a list of dict rows (the CSV columns). Quarter is `1`-`4` or `"a"` for
annual. Area FIPS: `US000` national, `11000` = DC, `11001` = a county, `C####` = metro.
### Caching, errors, and retry
```python
client = BLSClient(API_KEY, cache=True, retries=2, backoff=0.5)
client.fetch(...) # identical requests served from ~/.cache/bls (1-day TTL)
client.queries_used # network calls this session (cache hits excluded)
```
```python
from bls_client import BLSQuotaError, BLSRequestError
try:
client.fetch(ids, 2024, 2025)
except BLSQuotaError: # daily 500-query limit hit (resets midnight ET)
...
except BLSRequestError: # malformed request / bad series ID
...
```
Transient failures (timeouts, dropped connections, `429`/`5xx`) retry automatically with
exponential backoff and honor a `Retry-After` header. Quota rejections and 4xx are never
retried. Set `retries=0` to disable.
---
## 6. Coverage
| Survey | What's available |
|---|---|
| **LAUS** — local area unemployment | state / metro / county rates; DC-region dashboard |
| **CES** — payroll employment | national by supersector; state/metro |
| **CPS** — household survey | national unemployment rate, participation |
| **JOLTS** — job openings & turnover | openings, hires, quits, layoffs, separations; dashboard |
| **CPI** — consumer prices | all-items, core, food, energy, gasoline, shelter, …; dashboard |
| **PPI** — producer prices | all commodities, final demand, food, energy |
| **OES** — occupational wages | employment + wage percentiles by SOC; 15 named occupations |
| **ECI** — employment cost index | total comp / wages / benefits × civilian/private/gov |
| **ECEC** — employer cost levels | full breakdown: comp, wages, benefits, paid leave, supplemental, health insurance, retirement, legally-required |
| **Productivity & costs** | output/hr, unit labor cost, compensation, hours × business/nonfarm/manufacturing |
| **QCEW** — quarterly census | national/state totals **plus** full county/industry detail (CSV module) |
---
## 7. Testing
```bash
pip install -e ".[dev]"
pytest # 125 offline tests — lock every encoding; no network, no quota
pytest -m live # +live smoke tests against the real API (needs network + key)
```
The offline tests are the regression guard: they assert each builder/helper produces its
exact known-good series ID, so a re-broken encoding fails immediately and for free. The
live tests catch the other failure mode — BLS silently retiring a code (as happened with
the 2010→2018 SOC change).
---
## 8. Known limitations
- **Coverage is the headline cut** of each survey, not an exhaustive mirror (CES is
national supersectors + a couple of states; CPI is the common items; OES ships 15 named
occupations). Any SOC/industry still works through the parameterized builders — the
pre-built helper dicts just cover the common asks. Widen them as needed.
- No rate-limit *pacing* beyond caching + retry; a heavy batch can still approach the
500/day quota (the `queries_used` counter and `BLSQuotaError` make that visible).
---
## 9. Change history
| Commit | What |
|---|---|
| `4fe7343` | Fixed all broken series-ID encodings (JOLTS/OES/ECI/productivity/PPI/SOC); env-var config; project README |
| `881d62c` | `pyproject.toml` packaging + pytest suite |
| `9e2c55c` | QCEW CSV module, ECEC benefit breakdown, response caching + typed errors |
| `0d0d6da` | Transient-network retry with exponential backoff |