Tests. And typing. And comments.
Tests can help you be sure that each piece is working as intended and that they’re working together. You can also mock bits out or create a temporary database for local dev.
Adding typing and comments, especially docstrings will help others read your code more easily.
Yes!
What’s the format for talks? (Maybe I just missed it, but I didn’t see a description)
I’ve seen too many issues caused by relying on the system python. For starters, it’s much more difficult to fix if you screw something up.
I tell everyone to use pyenv – but only to install specific versions of python. I then create a primary venv for a couple of versions, and that’s what I use as my “system” version. Each project creates a separate venv based on the pyenv versions. I use autoenv to activate and deactivate as needed.
It’s a little more setup, but it avoids so much magic, and it’s extremely robust.
I use DictReader all the time, along with a library to hande the type detection. This is the way to go, especially if you need to process line by line or filter columns and rows out first.
Regardless, I’ll avoid pandas wherever I can. It’s not something I want in production level code if I can help it.
Also, you probably want a rotating file handler for the logging.