Case · Backend · Python
csv-doctor: CSV profiling and cleaning
A CSV check-and-clean service in Python (FastAPI, no pandas). Upload a file up to 5 MB: the encoding (utf-8, cp1251, koi8-r, UTF-16 with BOM) and delimiter are detected, you get a column profile and a list of problems with line numbers and hints, then download a cleaned CSV. Dangerous cells (=, +, -, @) are caught as CSV injection. The file is parsed in memory and never stored.
- Python
- FastAPI
- uvicorn
- hypothesis
- mypy --strict
- ruff
- pip-audit
- Docker
- GitHub Actions
01 · How it looks
Screenshots
02 · About
What it is and why
A CSV check-and-clean service in Python (FastAPI, no pandas). Upload a file up to 5 MB: the encoding (utf-8, cp1251, koi8-r, UTF-16 with BOM) and delimiter are detected, you get a column profile and a list of problems with line numbers and hints, then download a cleaned CSV. Dangerous cells (=, +, -, @) are caught as CSV injection. The file is parsed in memory and never stored.
Limitations
The image is 202 MB: mostly the python:3.12-slim base.
Memory peaks near 115 MiB when cleaning a 3.8 MB file; at most four parses run at once.
About 1,100 requests per second was measured with the client on the same machine: a stability check, not a benchmark.
The project runs on a small VPS with demo data: no redundancy or backups.
What is inside
- Encoding and delimiter autodetection, a column profile, outliers by 1.5·IQR, ambiguous dates
- CSV-injection protection on input (an error) and on cleaning output (an apostrophe); +7 phone numbers are left alone
- 43 tests; hypothesis: an encoding round trip, idempotent cleaning clean(clean(x)) == clean(x)
- CI: ruff, mypy --strict, pytest --cov, pip-audit over a hash-pinned requirements.lock; a 3.8 MB file (95,000 rows) is parsed in 1.5 s
03 · How to run
Run it locally
git clone <repository on request> csv-doctor
cd csv-doctor
python -m app04 · Code
The repository is private
The source is in a private repository. Message me on Telegram and I will open access.
Request the code