Welcome to our Ask an Agodan series, where we bring you closer to the people powering our global team. At Agoda, we’re building a community of innovators and problem-solvers to bridge the world through travel. This series will give you an exclusive glimpse into how we make it happen. In our previous editions, we covered moving to Bangkok for work and landing a job in Tech at Agoda.
Today, we’re excited to introduce Shaun Sit, a Director of Engineering at Agoda. Shaun spearheads our Data Platform org, where we manage over 30PB of data to help our employees make better decisions and drive Agoda’s growth. He also leads our GPT Tooling Team, developing GenAI-powered tools to supercharge internal productivity across the company.
We invited you to send in your questions about Tech at Agoda, and now we’re back with Shaun’s answers. Thank you to everyone who sent in questions! Be sure to follow our social media channels to stay in the loop about future editions of Ask an Agodan, where you’ll have more chances to connect with our talented team and get your questions answered.
Agoda’s Data Platform and Tech Stack
You asked: What tech stack does Agoda use?
Our data platform is hosted on-premise in one of our four data centers. We primarily use Apache Spark for data processing and run jobs on YARN. We also have plans to transition to Kubernetes.
For storage, we rely on Vast Storage, an object store provider. Currently, we manage job scheduling with Oozie, but we are planning a migration to Kubeflow. For querying, we use Impala and are moving towards StarRocks. We use Vertica for specific cases that require fast query response times, such as critical reports. Additionally, we also leverage Kafka as our message broker, handling trillions of events daily.
For data quality, we use a combination of in-house tools and Quilliup. For data discoverability, we rely on OpenMetadata. Additionally, we have developed a custom low-code/no-code platform called Workflow Editor, which allows us to build complex data pipelines. Core transformations are handled using SparkSQL. We also have our own notebook solution called Zoidberg, designed for interactive data exploration and analysis.
You asked: Can you share some of the technologies and concepts Agoda uses to manage big data while ensuring performance and scalability?
We invest heavily in building custom tooling to help us manage our large-scale data platform. This includes developing diagnostic tools and automatic remediation solutions for known cluster issues, as well as maintaining a robust set of monitoring systems to ensure smooth operations. Alerts are set up so that our on-call engineer(s) can quickly react to issues, such as a well-meaning team deploying a job that unintentionally impacts platform health.
In terms of concepts, to ensure scalability and agility, we take an opinionated approach to how teams run their jobs on our clusters. We provide tools and internally developed libraries that teams must use to develop and deploy jobs interacting with data. This approach minimizes disruption when we make changes to the underlying infrastructure or tech stack. For example, by encouraging teams to use our workflow editor, we ensure that future migrations or upgrades are transparent to them, allowing us to move quickly without impacting productivity.
You asked: What kinds of GenAI tools has Agoda developed or implemented to support the software development process?
At Agoda, we’ve adopted an inside-out approach to GenAI, focusing first on internal productivity tools before gradually scaling out to customer-facing applications. Currently, we have over 120 services that leverage GenAI in some form. Here are some of the GenAI tools we have that support the development lifecycle:
- Planning:
GenAI tools help with better task descriptions for scrum stories. Additionally, assistants help with meeting summarization and act as collaborative chat assistants in online discussions, ensuring key points are captured and shared efficiently.
- Development:
Assistants allow developers to query our knowledge repository for quick answers to technical questions. Additionally, we have GenAI-powered tools like sketch-to-UI code used in prototyping, tools that automate code migrations (addressing tech debt), and tools to auto-generate README documentation for repositories. GitHub Copilot is also widely used for code completion in day-to-day development.
- Merge Requests (MR):
We have a GenAI-powered bot that assists with code reviews and linting, ensuring code adheres to best practices before merging.
- Post-Deployment & Maintenance:
After deployment, more GenAI tools assist with incident management by identifying patterns in error logs and finding similar incidents. We also have chat assistants to help with automated support ticket resolution, speeding up the process of addressing issues.
By integrating GenAI throughout the software lifecycle, we ensure that our developers can work more efficiently, from planning and development to post-deployment maintenance.
Handling Big Data at Scale at Agoda
You asked: I’m curious—how does Agoda manage petabytes of data to fuel its growth?
Managing tens of petabytes of data is a challenge that requires both scalable technologies and a thoughtful approach. At Agoda, we build tools that give teams visibility into their storage and compute costs, ensuring they are good stewards of the company’s resources. Regular capacity planning helps us stay ahead of data growth, and while our on-premise infrastructure doesn’t have the immediate elasticity of the cloud, we believe our approach balances scalability, performance, and cost-effectiveness. That being said, we’re always looking to improve our data management tools to ensure efficient use of resources—it’s an ongoing process that we don’t expect to ever stop.
You asked: How much traffic does Agoda handle?
Agoda manages a substantial volume of data. We store ~30+ PB of data, with ~600 TB of new data written daily. Our platform processes approximately ~2.8 trillion events through Kafka and runs around 100K Spark jobs every day to handle data processing at scale.
Career Tips for Joining Tech at Agoda
You asked: I’m applying for a software engineer role at Agoda. Do you have any tips for someone looking to join the Agoda team?
We are always on the lookout for great talent. When assessing candidates, we consider a combination of relevant experience, technical skills, and cultural fit. We especially value qualities like a strong sense of ownership and technical expertise.
If I had to offer some advice beyond that, it would be to cultivate a deep curiosity—dive deep into the details of how technologies work. Additionally, having a strong desire to help others is key. You often grow as an engineer when you assist others with their technical challenges, and that collaborative mindset is something we highly appreciate.
Check out these blogs for more useful tips:
- Interview Guide for Tech Candidates
- Ask an Agodan: How do I get a Tech job at Agoda?
- Interviewing at Agoda: What to ask your interviewer
- Interviewing at Agoda: Tackling tricky questions
- How We Hire: A Guide to Agoda’s Hiring Process
I’m passionate about solving real-world problems using a data-driven approach and would love to join your team. How can I make that happen?
To maximize your chances of joining the Agoda team, it’s best to focus on three key areas: relevant experience, technical expertise, and cultural fit.
- Relevant Skillset & Technical Expertise: Having the right skillset is important. Within the data platform team, we look for individuals with experience in our tech stack (or equivalent technologies). Familiarity with tools like Apache Spark, Kafka, and Kubernetes, or experience with large-scale data platforms, will significantly improve your chances.
- Cultural Fit: Beyond technical skills, we value qualities like a strong sense of ownership, effective communication, and a collaborative mindset. As I mentioned earlier, having a curiosity to dive deep into how things work and a willingness to help others with their technical challenges are qualities we highly appreciate.
By focusing on these areas, you’ll be well-positioned to join our team and contribute to solving real-world problems with a data-driven approach.
What key skills or knowledge should an entry-level software developer have to join Agoda?
We do most of our associate developer hiring through our internship alumni. If you’re still studying and interested in joining Agoda, I strongly encourage you to apply for our internship program. Internships are a great way to gain hands-on experience, get a feel for what it’s like to work in Tech at Agoda, and understand our company culture.
For associate developers, we look for individuals who can think quickly on their feet and keep up with the fast pace at Agoda. The ability to learn quickly, including picking up new programming languages, is crucial. Additionally, effective communication skills are important, as collaboration is key to success in our teams.
Learn more about Agoda’s paid internship program here.
Join our Tech team and be part of building travel for the world. We’re hiring!
