The March Analyst

Requirements, basic design decisions, and technology choices.

Create first gRPC service; will just echo for now.

Start logging and reporting

Create Linux service for scraper. Shop for DBs. Setup Prometheus.

Change db to write-then-read with cache before returning response. Later will have a different worker do this.

Pause implementation to consider different API patterns.

After a break, added some extra documentation and tweaked some existing commands.

Create backend worker. Played more with logging, trying fluent-bit. Switched from LevelsDB to sqlite to fix parallel access.

Implemented proxy pools. Switch from expiration to freshness pattern.

Migrate everything over to docker. Use an ELK stack.

Switched from sqlite to file directory. Added compression.