The first article was about moving decoding out of the API. That was the easy part.
But after a few months, I realized the async queue only solved half the problem.
The Phase 1 Architecture (Still Had Issues)
After decoupling from the API, the flow looked like:
Mobile App → Backend API → Queue → Backend Worker → Python Decoder → Database
The backend worker now handled all the heavy lifting. Decoding, parsing, generating outputs, writing to the database.
Sounded good on paper.
In production, it was a mess.
What Was Actually Wrong
The backend worker contained a Python runtime. Inside a NestJS/TypeScript deployment.
Every backend deploy shipped the decoder. Every decoder change required a backend release. When the Python environment broke, the backend workers crashed. Scaling one meant scaling the other.
The fundamental issue: I was treating a data processing problem like an API request problem. They're not the same.
The backend should do one thing: orchestrate requests and manage data. Not run arbitrary Python code or generate CSV files.
The Phase 2 Solution: Service Boundaries
I extracted the decoder into its own service.
Now the architecture is:
Backend
↓ (creates job)
Queue
↓
Backend Worker
↓ (enqueue to decoder)
Decoder Service API
↓
Decoder Worker
↓
Python Decode Process
↓ (generates artifacts)
Artifacts (CSV/JSON)
↓ (upload)
Blob Storage
↓
Backend Artifact Ingestion
↓
Sensor Tables
The key shift: the backend doesn't decode anymore. It orchestrates.
Why This Matters
Async queues aren't enough.
A lot of engineers stop at Phase 1. API → Queue → Worker. But if the worker is still doing heavy lifting and coupled to your backend, you've just moved the problem, not solved it.
Service boundaries matter more than queue design.
Queues solve request latency. They don't solve:
- Ownership (who deploys the decoder?)
- Runtime isolation (why is Python inside TypeScript?)
- Scaling independence (why does the decoder scale with the API?)
- Technology isolation (they shouldn't share dependencies)
Different workloads deserve different services.
Backend workload:
- Handle HTTP requests
- Validate data
- Manage database consistency
- Orchestrate workflows
Decoder workload:
- Process binary sensor data
- Generate multiple output formats
- Handle large file operations
- Run heavy computation
Those are different problems. Different languages. Different dependencies. Different scaling patterns.
Keeping them together creates operational debt.
The Artifact Contract
Instead of the decoder returning raw rows, I made it generate artifacts.
ppg.csv (heart rate data)
eda.csv (skin conductance)
bioz.csv (bio impedance)
summary.json (metadata)
The backend doesn't care how decoding works. It doesn't understand the raw sensor format. It just:
- Enqueues a decode job
- Waits for completion
- Downloads the artifacts
- Ingests them into the database
This creates a contract between services. The decoder owns the data format. The backend owns the schema.
That's a much stronger boundary than passing objects between processes.
Independent Scaling
Before, if I needed to handle more decode jobs, I had to scale the backend workers. Which meant scaling the API workers too. Everything moved together.
Now:
Backend API: 2 pods
Backend Worker: 2 pods
Decoder Service: 1 pod
Decoder Worker: 10 pods
The decoder can auto-scale independently. An API traffic spike doesn't affect decode throughput. A decode backlog doesn't affect API latency.
That's the whole point.
The Messy Production Reality
The architecture looked clean until I tried to run it end-to-end.
Real problems I hit:
- Azure blob downloads would randomly abort mid-transfer
- Artifact manifest files went missing
- Container filesystem permissions were wrong
- Worker pods weren't scheduled correctly
- Queue consumers scaled to zero unexpectedly
- Service-to-service communication timeouts
- Retry logic didn't work the first three times
Those are classic distributed systems problems. They don't show up in local testing. They appear when:
- Real NAND file (150+ MB)
- Uploaded to blob storage
- Backend queues a decode job
- Decoder service pulls the file
- Python processes it
- Artifacts uploaded back to blob
- Backend downloads artifacts
- Data ingested into database
152,504 PPG rows. 9,344 EDA rows. 17,872 BIOZ rows.
All successfully processed end-to-end.
That's when I knew the architecture actually worked.
The Real Lessons
Async isn't the same as decoupled.
Async moves work off the critical path. But if the work is still tightly coupled to your backend, you haven't actually decoupled anything.
Service boundaries are about ownership.
A proper service boundary means:
- One team can deploy it independently
- Changes don't affect other services
- Failures are isolated
- Scaling decisions are independent
The decoder service doesn't care about the backend. The backend doesn't care how decoding works. That's a healthy boundary.
Technology choices should align with the problem.
Python is great at data processing. It's terrible as an embedded runtime inside a Node.js service. TypeScript is great for API servers. It's not great for scientific computing.
Using the right tool for each job simplifies everything.
Contracts matter more than shared code.
Instead of the decoder and backend sharing libraries or passing objects, they communicate through artifacts. CSV files and JSON. Language-agnostic. Version-independent.
If the decoder wants to change its implementation, it doesn't touch the backend.
The Evolution
Phase 1: Move processing off the API request path Phase 2: Separate service boundaries based on workload type Phase 3 (probably): Each service has its own deployment, monitoring, scaling
That's the trajectory from monolith to distributed system.
It's messy. It requires solving problems that don't exist in monoliths. But when it works, you get something much stronger.
A system where each component has one job. Scaling independent. Deployment independent. Failure isolated.
That's worth the complexity.