Think about facing a tricky problem with no easy solution. That’s what engineers in big tech have to deal with all the time. IDuring an Expert Edge session, we had the chance to talk to Swapnil Agarwal, a senior software engineer with over ten years of experience, to learn how he and his team solve these tough problems. Swapnil shared some great ideas about how engineers break down big challenges and find smart, simple ways to fix them. Let’s take a look at what he had to say about solving problems in the world of big tech.
Understand the Problem
The first thing we need to ask ourselves is, who are we solving this for? What is the use case? This allows us to ensure the solution we design solves the actual need. Engineers often dive into complex solutions or technologies they’re passionate about, but it’s crucial to avoid over-engineering. Sometimes the simplest solution, if thought through, is the best one.
Break Down Complexity
When faced with large-scale challenges, start with high-level ideas and gradually refine them. For example, while building a data sync mechanism, we prioritized requirements, designed generic systems, and focused on simplicity before adding complex features like real-time syncing.
Prepare for Failures
Failures are inevitable, so design systems with this in mind. Ask yourself, what could go wrong? Train your brain to anticipate potential issues and plan for them. Create mechanisms to handle issues like executor failures, network disruptions, and scheduling conflicts. Implementing checkpoints and robust error-handling ensures that your systems remain resilient and adaptable in the face of challenges.
Optimize Gradually
Premature optimization, which is something everyone enjoys working on, should be delayed as much as possible. First, ensure the system works correctly. Then focus on performance and scalability. For example, in data sync, instead of downloading everything repeatedly, optimize by only fetching changes, reducing the load significantly.
Build for Scale
Scalability and supportability are key. Establish processes for monitoring and debugging. Create tools and documentation for support teams to troubleshoot issues independently, reducing reliance on developers.
Secure Systems
Security and privacy are non-negotiable. Implement controls to prevent data leaks, both accidental and intentional. Ensure processes like approval mechanisms are in place for accessing sensitive information.
On AI and Coding
AI won’t replace developers but will handle redundant tasks. Developers need to operate at a higher level of abstraction, focusing on system design and optimization while letting AI handle mundane coding.
Advice for Engineers
Stay curious and be passionate about solving problems. Learn the fundamentals deeply, whether it’s databases, APIs, or distributed systems. Use failures as a learning opportunity and never stop upskilling. And most importantly, focus on the user’s needs.