Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
As a developer with undergraduate-level programming experience and a decade’s experience teaching computer science and programming at GCSE and A Level, I had an ambitious idea for a project that involved web scraping. The concept was solid: develop a system to manage proxy servers for reliable data collection. However, the implementation requirements quickly exceeded my personal programming expertise. This is my story of addressing technical debt with AI assistance.
Enter Claude, Anthropic’s AI assistant. I chose to work with Claude specifically because Anthropic’s commitment to ethical AI development aligns with my own principles. More practically, Claude’s unique ability to handle complex software architecture and generate structured artifacts made it an ideal programming partner.
The collaboration started promisingly. I provided the vision and requirements, Claude helped with implementation details, and together we built a working proxy management system. Claude could generate code, tests, and documentation, while I reviewed, validated, and steered the development direction.
One particularly valuable feature was Claude’s ability to generate “artifacts” – structured blocks of code, documentation, or other content that could be directly used in the project. This capability proved essential for maintaining consistency and quality throughout the development process.
Despite this promising start, I made a crucial mistake. While I was directing the project and making decisions, I wasn’t fully embracing the role of software architect. Instead, I was falling into a pattern of:
This approach worked fine initially. The code worked. The tests passed. We were shipping features and solving problems. But then came the bug that would change everything – a seemingly simple issue that would require 70+ updates to fix, teaching us valuable lessons about software architecture along the way.
Our proxy manager started simple enough:
The initial implementation appeared sound:
class ProxyManager:
def __init__(self):
self.proxies = []
self.stats = {}
def add_proxy(self, proxy):
self.proxies.append(proxy)
self.stats[proxy.id] = {"success": 0, "failure": 0}
def get_next_proxy(self):
# Simple rotation logic
return self.proxies.pop(0)
Then came the requirement: “We need to track detailed proxy health metrics and adjust rotation based on performance.”
Simple enough, we thought. We added health tracking:
def update_health(self, proxy_id, success, response_time):
stats = self.stats[proxy_id]
stats["success" if success else "failure"] += 1
stats["response_times"].append(response_time)
stats["health"] = self._calculate_health(proxy_id)
self._update_rotation_order() # This seemed harmless at the time
This is where things started to unravel. The seemingly innocent _calculate_health
method needed to know about:
While _update_rotation_order
needed to understand:
Every component became entangled with every other component.
What followed was a nightmare of cascading changes. Here’s a real excerpt from our git log:
fix: health calculation for edge case
fix: rotation order after health update
fix: proxy availability check timing out
fix: health score normalization affecting rotation
fix: rotation strategy breaking health monitoring
fix: proxy validation timeout causing incorrect health
fix: health score calculation timeout
fix: unexpected proxy removal during health check
Each fix introduced new problems because our monolithic ProxyManager was violating the Single Responsibility Principle in multiple ways:
After two weeks and 70+ updates, we had our epiphany. The problem wasn’t with any individual piece of code. The problem was architectural. Our ProxyManager was trying to be everything to everyone.
Looking at our CHANGELOG.md was enlightening:
## [v0.4.5] - 2024-11-29
### Addressing
- Implementing actual request logic in measure_latency
- Adding proxy authentication support
- Adding proper timeout handling
## [v0.4.4] - 2024-11-29
### Addressing
- Adjusting health score weighting to prioritise response time
- Attempting to achieve correct test score via 3:1 time weighting
## [v0.4.3] - 2024-11-29
### Addressing
- Simplifying health score calculation to use basic average
- Attempting to achieve consistent 0.75 score in test conditions
We were treating symptoms, not the disease. The disease was our monolithic architecture.
The breakthrough came when we started thinking in terms of atomic components. Each piece should have one clear responsibility. This led to a clean, maintainable architecture:
proxy_manager/
├── entities/
│ ├── proxy.py # Core proxy representation
│ ├── proxy_stats.py # Statistics tracking
│ └── proxy_health.py # Health status
├── health/
│ ├── validator.py # Validation logic
│ └── stats.py # Statistics management
└── pool/
├── storage.py # Proxy storage
└── rotation.py # Rotation logic
Each component had:
This experience led us to create the AI Architect Kit, a template and framework for solo developers, leveraging AI assistants when building maintainable systems from the start. It embodies everything we learned about:
The time you invest in proper architecture pays back manifold in maintenance and development speed.
In the next post, we’ll explore the principles we established to prevent these issues, including:
This is part 1 of a 5-part series on clean architecture in AI-assisted development.
Subscribe to get the latest posts sent to your email.