Webscraper 1
Requirements, basic design decisions, and technology choices.
2024LinkWebscraper 2
Create first gRPC service; will just echo for now.
2024LinkWebscraper 5
Change db to write-then-read with cache before returning response. Later will have a different worker do this.
2024LinkWebscraper 8
Create backend worker. Played more with logging, trying fluent-bit. Switched from LevelsDB to sqlite to fix parallel access.
2024LinkWebscraper 9
Implemented proxy pools. Switch from expiration to freshness pattern.
2024Link