Fundamentals of Data Science Project Sample: Census Data Analysis in Python

How Do You Clean Census Data Without Distorting It?

Audit first, fix second, and flag every value you change. Imputing before you have counted the damage hides how much of your result rests on guesses (Little and Rubin, 2019).

Issue	Check	Fix
Stray spaces, mixed case	value_counts() on each text column	Trim, collapse spaces, title-case
Exact duplicate rows	duplicated().sum()	Drop and report the count
Impossible ages (below 0 or above 110)	Range test	Set to missing, then impute
Missing ages	isna().sum()	Median age within the relationship group, with an age_imputed flag
Blank infirmity, religion, marital status, occupation	isna().sum()	“None” for infirmity, “Not Stated” elsewhere

Keep category labels as text. Swapping them for integers during cleaning makes later charts unreadable and invites silent filter errors. Encode only inside a modelling pipeline, if you build one (Wickham, 2014).

Print the missing-value audit before and after the fix, and put both in the report. A marker can then see exactly what changed. The first block below uses pandas, NumPy and Matplotlib (McKinney, 2022; Harris et al., 2020; Hunter, 2007).

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
# 1. Load as text so nothing is coerced silently, then normalise the headers
df = pd.read_csv('census.csv', dtype=str, keep_default_na=False)
df.columns = [' '.join(c.split()).lower().replace(' ', '_') for c in df.columns]
required = {'house_number', 'street', 'first_name', 'surname', 'age',
 'relationship_to_head_of_house', 'marital_status', 'gender',
 'occupation', 'infirmity', 'religion'}
missing_cols = required - set(df.columns)
assert not missing_cols, f'Rename these headers to match your file: {missing_cols}'
# 2. Standardise text: trim, collapse spaces, consistent case, blanks -> NaN
text_cols = [c for c in df.columns if c != 'age']
for c in text_cols:
 df[c] = df[c].str.split().str.join(' ').str.title().replace({'': np.nan})
df['age'] = pd.to_numeric(df['age'].str.strip(), errors='coerce')
# 3. Audit BEFORE fixing, so the report can show what changed
print(df.isna().sum())
print('Exact duplicates:', df.duplicated().sum())
print('Impossible ages:', ((df['age'] < 0) | (df['age'] > 110)).sum())
# 4. Fix with stated rules; keep a flag on every imputed age
df = df.drop_duplicates()
df.loc[(df['age'] < 0) | (df['age'] > 110), 'age'] = np.nan
df['age_imputed'] = df['age'].isna()
df['age'] = df['age'].fillna(
 df.groupby('relationship_to_head_of_house')['age'].transform('median'))
df['age'] = df['age'].fillna(df['age'].median()).round().astype(int)
df['infirmity'] = df['infirmity'].fillna('None') # blank = none recorded
df = df.fillna({c: 'Not Stated' for c in text_cols if c != 'infirmity'})
assert df.isna().sum().sum() == 0, 'Unresolved missing values remain'
print('Imputed ages:', int(df['age_imputed'].sum()), 'of', len(df))
# 5. Derived fields used by every later step
df['household'] = df['house_number'] + ' ' + df['street'] # dwelling = number + street
df['age_band'] = pd.cut(df['age'], bins=[-1, 4, 15, 17, 24, 39, 64, 120],
 labels=['0-4', '5-15', '16-17', '18-24', '25-39', '40-64', '65+'])

Task A: What Should the Vacant Plot Become?

Each option gets one indicator, one reason that indicator is fair, and one honest weakness.

High-density housing

Density brings agglomeration gains and congestion costs (Duranton and Puga, 2020), and younger households usually accept the congestion more readily. The indicator is the share of residents aged 18–39. Support it with a population pyramid and persons per dwelling. Weakness: age shows who could live in flats, not who wants to.

Low-density housing

The indicator is the share of dwellings with at least one child under 16, a rough measure of family homes. Cross-check it against the 65+ share. Both housing options compete for the same plot, so the ranking breaks ties by the share of residents affected. Weakness: plot size and build cost are not in the data.

Train station

With no university in town, student residents must travel. Count students aged 18+ as the lower bound for commuters, then count all working-age adults who are not retired as the upper bound. Report the range. Recommend a station only if the lower bound clears its threshold on its own. If only the upper bound does, say the census cannot settle it. Weakness: the census does not record where anyone works.

Religious building

Count residents by faith, drop “None” and “Not Stated”, and take the largest remaining group as a share of everyone. Run value_counts() first. If “None” outnumbers every faith, that points to a secular majority, not to zero demand. The census does not show existing places of worship, so state that you assume the plot is the only candidate site.

Emergency medical building

Emergency demand tends to cluster at the age extremes and among people with a recorded infirmity, so the indicator combines residents aged 0–4, aged 65+ or with an infirmity. Report women aged 18–39 separately as maternity-age context, labelled a proxy because the census records no pregnancies. Weakness: infirmity categories differ widely in how much emergency care they imply.

Task B: Where Should the Council Spend Its Budget?

Employment and training

The unemployment rate is unemployed residents divided by an economically active proxy: aged 16–64, not a student, not retired. Check df['occupation'].value_counts() for the exact unemployed label before filtering. Plot the rate by age band, not by single year. Single years produce noisy spikes that tell a reader nothing.

Old age care

Use the share of residents aged 65+, plus the share of that group living alone. Lone elderly residents are the sharper signal, because nobody in the house can provide day-to-day care.

Schooling

Use the share aged 5–15, and adjust the range to the school-age definition in your brief. Weakness: the census holds no school capacity data, so the indicator measures pressure, not shortfall.

General infrastructure

Use the share of dwellings with six or more occupants, and houses per street as a load indicator for roads and utilities. Count dwellings by house number and street together. Counting house numbers alone confuses a number shared by several streets with one home.

How Do the Options Rank Against Each Other?

Rank by score first, then by the share of residents affected. The second block computes every indicator, scores it against the thresholds, stress-tests the winner and saves two charts. Run it in the same notebook as the first block.

# 6. Shared helpers: every indicator is a percentage, never a raw count
N = len(df)
share = lambda mask: round(100 * mask.sum() / N, 1)
hh_size = df.groupby('household').size() # occupants per dwelling
is_student = df['occupation'].str.contains('student', case=False)
is_retired = df['occupation'].str.contains('retired', case=False)
is_unemployed = df['occupation'].str.contains('unemploy', case=False)
# 7. Task A indicators
faith = df.loc[~df['religion'].isin(['None', 'Not Stated']), 'religion'].value_counts()
A = {
 'High-density housing': share(df['age'].between(18, 39)),
 'Low-density housing': round(100 * df.loc[df['age'] < 16, 'household'].nunique() / len(hh_size), 1),
 'Train station': share(is_student & (df['age'] >= 18)),
 'Religious building': round(100 * faith.iloc[0] / N, 1),
 'Emergency medical building': share((df['infirmity'] != 'None') | (df['age'] >= 65) | (df['age'] <= 4)),
}
commuter_upper = share(df['age'].between(16, 64) & ~is_retired) # upper bound for the station
# 8. Task B indicators
active = df['age'].between(16, 64) & ~is_student & ~is_retired # economically active proxy
lone_elderly = (df['age'] >= 65) & (df['household'].map(hh_size) == 1)
B = {
 'Employment and training': round(100 * (is_unemployed & active).sum() / active.sum(), 1),
 'Old age care': share(df['age'] >= 65),
 'Schooling': share(df['age'].between(5, 15)),
 'General infrastructure': round(100 * (hh_size >= 6).sum() / len(hh_size), 1),
}
print('Station range, lower vs upper (%):', A['Train station'], commuter_upper)
print('Lone elderly as % of residents 65+:', round(100 * lone_elderly.sum() / (df['age'] >= 65).sum(), 1))
# 9. Score against thresholds declared in advance: (medium, high)
thresholds = {
 'High-density housing': (20, 35), 'Low-density housing': (30, 50),
 'Train station': (5, 10), 'Religious building': (20, 40),
 'Emergency medical building': (15, 30),
 'Employment and training': (5, 10), 'Old age care': (10, 20),
 'Schooling': (15, 25), 'General infrastructure': (10, 25),
}
def score(name, value, shift=0):
 medium, high = thresholds[name]
 return 2 if value >= high + shift else 1 if value >= medium + shift else 0
def rank(indicators):
 out = pd.DataFrame({'indicator_pct': pd.Series(indicators)})
 out['score'] = [score(n, v) for n, v in out['indicator_pct'].items()]
 return out.sort_values(['score', 'indicator_pct'], ascending=False) # ties: share affected
print(rank(A))
print(rank(B))
# 10. Stress test: does the winner change if every threshold moves by 5 points?
def winner(indicators, shift):
 return max(indicators, key=lambda n: (score(n, indicators[n], shift), indicators[n]))
for shift in (-5, 0, 5):
 print(f'Shift {shift:+d} | Task A: {winner(A, shift)} | Task B: {winner(B, shift)}')
# 11. One chart per task
fig, axes = plt.subplots(1, 2, figsize=(12, 4))
for ax, (title, data) in zip(axes, [('Task A: land use', A), ('Task B: spending', B)]):
 ax.barh(list(data), list(data.values()), color='#2b6cb0')
 ax.set_xlabel('% of residents or dwellings affected')
 ax.set_title(title)
plt.tight_layout()
plt.savefig('decision_charts.png', dpi=200)
plt.show()
# 12. Population pyramid for the housing and care arguments
fig, ax = plt.subplots(figsize=(6, 5))
edges = np.arange(0, 105, 5)
for sex, colour, sign in [('Male', '#2b6cb0', -1), ('Female', '#c05621', 1)]:
 counts, _ = np.histogram(df.loc[df['gender'] == sex, 'age'], bins=edges)
 ax.barh(edges[:-1], sign * counts, height=4.5, align='edge', color=colour, label=sex)
ax.xaxis.set_major_formatter(lambda x, _: f'{abs(int(x))}') # show positive counts on both sides
ax.set_xlabel('Residents')
ax.set_ylabel('Age (5-year bands)')
ax.legend()
plt.tight_layout()
plt.savefig('pyramid.png', dpi=200)
plt.show()

Paste the printed output into the two tables below. Then write one paragraph per task naming the winner, the runner-up and the threshold shift at which the ranking changes.

Task A decision matrix

Option	Indicator (%)	Score (0–2)	Decision
High-density housing	[value]	[value]	[value]
Low-density housing	[value]	[value]	[value]
Train station	[lower]–[upper]	[value]	[value]
Religious building	[value]	[value]	[value]
Emergency medical building	[value]	[value]	[value]

Task B decision matrix

Option	Indicator (%)	Score (0–2)	Decision
Employment and training	[value]	[value]	[value]
Old age care	[value]	[value]	[value]
Schooling	[value]	[value]	[value]
General infrastructure	[value]	[value]	[value]

What Are the Limits of This Analysis?

No census analysis is better than its proxies, so say where yours is weakest.

• One snapshot. The census shows who lives in the town now, not how the population is changing.
• Proxies, not measurements. Age stands in for housing preference, students stand in for commuters, and infirmity stands in for emergency demand.
• Judgement-based thresholds. The stress test shows how much the answer depends on them.
• Mock data. Nothing here transfers to a real town without real planning data on land, cost and existing services.
• Personal data. If you repeat this on real records, remove names and exact addresses first. UK GDPR expects data minimisation, and a shared notebook is a disclosure.

Closing Recommendation

A strong final paragraph does four things: it names the option, quotes the indicator, states the threshold it cleared, and admits the weakest assumption. For example: “Build [option] on the vacant plot. [Indicator] reaches [value]%, against a high threshold of [value]%. The ranking holds under a five-point shift, but it rests on the assumption that [weakest assumption].”

Need a worked example for your own dataset? Talk to [Brand] about an original, step-by-step data science solution.

References

• Duranton, G. and Puga, D. (2020) ‘The economics of urban density’, Journal of Economic Perspectives, 34(3), pp. 3–26.
• Harris, C.R. et al. (2020) ‘Array programming with NumPy’, Nature, 585(7825), pp. 357–362.
• Hunter, J.D. (2007) ‘Matplotlib: a 2D graphics environment’, Computing in Science & Engineering, 9(3), pp. 90–95.
• Little, R.J.A. and Rubin, D.B. (2019) Statistical analysis with missing data. 3rd edn. Hoboken: Wiley.
• McKinney, W. (2022) Python for data analysis. 3rd edn. Sebastopol: O’Reilly Media.
• Wickham, H. (2014) ‘Tidy data’, Journal of Statistical Software, 59(10), pp. 1–23.