Cleaning 311 Request Data With AI: Duplicate Detection and Category Drift

311 Data Contains More Than a List of Complaints

A 311 export may include request identifiers, submission dates, descriptions, service categories, geographic coordinates, status updates, and closure dates. These fields describe different parts of a service interaction. A resident's original report records a concern, the category reflects an administrative classification, and the status describes what happened afterward. Treating all three as interchangeable can produce misleading summaries of neighborhood conditions.

For example, ten reports about a leaking hydrant might include repeated submissions from one address, reports about separate hydrants, and follow-up requests concerning an existing case. A simple count of ten does not establish ten separate maintenance problems. Before using AI tools for local government data, identify the meaning of each field and determine whether the export represents individual submissions, unique service cases, or a mixture.

• Retain the original request identifier.
• Preserve submission and update timestamps separately.
• Identify the distinction between requests and service cases.
• Keep original descriptions unchanged.
• Document the export date and geographic coverage.

Define Duplicate Detection Before Asking a Model

Duplicate detection begins with a rule about what counts as the same request. Identical request identifiers are straightforward administrative duplicates. Similar descriptions, nearby coordinates, and close submission times are weaker signals. Two residents can report the same broken streetlight, while one resident can report two different streetlights using almost identical wording. A language model can help identify these possibilities, but similarity does not establish that two records should be merged.

A practical workflow uses several comparison fields. Consider two reports submitted within 48 hours, with descriptions mentioning a blocked storm drain and coordinates within a short distance. These records may concern the same problem, but the distance threshold should reflect local geography and the type of service. A blocked drain and a damaged sidewalk can occur at the same location without being duplicate requests.

Ask the model to identify candidate pairs and explain the matching evidence. Require it to return the original identifiers, relevant dates, category values, and a short reason for the proposed match. Do not ask it to silently remove records. A human reviewer should decide whether the records represent one incident, separate concerns, or a follow-up that needs to remain linked to an earlier case.

• Exact matching for official identifiers.
• Text similarity for preliminary candidate discovery.
• Geographic proximity for location-based comparison.
• Submission timing for contextual review.
• Human decisions for merging or retaining records.

Category Drift Changes What a Dashboard Appears to Show

Category drift occurs when the meaning or use of a service classification changes over time. A city might rename a category, divide one category into several, or begin assigning a new label to requests that previously appeared under a general maintenance heading. The underlying resident concerns may remain similar while the administrative data changes. A chart that treats every label as a stable measurement can therefore make a service trend appear to rise or fall.

For example, imagine that reports about damaged sidewalks were previously classified under general street maintenance. A later administrative update introduces a dedicated sidewalk category. A dashboard might show a sudden increase in sidewalk requests after the change, even if the difference largely reflects the new classification. Compare category definitions and effective periods before interpreting the trend.

Create a category crosswalk that records old labels, new labels, their definitions, and the period in which each applied. When two categories cannot be meaningfully combined, preserve them separately and explain the break in the time series. A language model can suggest likely correspondences between labels, but staff should approve the mapping because category definitions belong to the agency maintaining the service system.

Use Language Models Without Rewriting Residents' Concerns

Language models are useful for finding recurring themes in free-text descriptions. They can suggest that reports mentioning overflowing bins, missed collection, and scattered trash may belong to a broader waste collection theme. This can help a civic tech volunteer group or a small planning office understand the vocabulary residents use, especially when the official categories are broad or inconsistent.

The danger is that a summary can flatten meaningful differences. A description about a dangerous crossing should not disappear into a general traffic concern category if the original submission describes a missing pedestrian signal. Preserve the original wording and add a separate normalized theme field. If the model cannot distinguish two themes confidently, use an uncertain or needs-review label instead of forcing a classification.

Public request descriptions may also contain names, phone numbers, addresses, or other personal details. Apply the municipality's data-handling rules before sending records to an external service. Use the minimum information needed for the task, follow applicable retention and access policies, and avoid uploading sensitive free text to a tool that has not been approved for that information.

• Keep source descriptions in a read-only field.
• Store AI-generated themes separately.
• Use a defined category vocabulary.
• Flag ambiguous classifications for human review.
• Follow local rules for personal and sensitive information.

Publish a Useful Service Picture, Not Just a Cleaner Spreadsheet

A cleaned 311 dataset can support neighborhood maps, service summaries, and questions about how public resources are distributed. Yet a high number of requests in one area may reflect greater reporting activity, a temporary event, easier access to the reporting system, or an actual concentration of service problems. A low count may indicate fewer problems, but it may also reflect barriers to submitting requests. Request volume alone is not a complete measure of service quality.

Show the time period, geographic unit, category definitions, and counting method alongside any chart. Distinguish new submissions from unresolved cases and completed work. If possible, retain separate measures for the number of reports, the number of unique cases, and the number of cases still open. This makes it easier for residents and staff to identify which administrative question a chart actually answers.

The same records can help identify places where social infrastructure needs attention. Repeated reports about broken lighting near a library, blocked access to a park, or damaged sidewalks around a community center may point to service issues affecting everyday public spaces. These connections are useful starting points for questions to the responsible departments, not automatic proof of unequal service or a substitute for local context.

The next step is to choose one service category, document its definition and date range, and produce a short table showing original request counts, possible duplicate groups, and unresolved classification issues. Share that table with the responsible 311 or public works office and ask who owns the category definitions, when they are updated, and how residents can request corrections.

Questions readers ask