Senior Engineer 2 – Inference Data Plane

Job Description:

• Act as a technical leader on the team, driving the end-to-end design, development, and delivery of critical data plane components hosting large generative AI models.
• Architect and refine system design proposals for our high-scale, multi-tenant AI inference cloud ecosystem, ensuring they meet rigorous availability and resiliency standards.
• Implement and optimize distributed inference hosting using techniques like tensor/data parallelism, KV cache optimizations, and smart routing.
• Work cross-functionally with Product Managers, customer-facing teams, and other engineering teams to align technical roadmaps with customer needs.
• Coach and mentor junior engineers, fostering a culture of technical excellence and continuous improvement.
• Maintain and operate critical, high-scale services, utilizing observability tools and defining SLOs to ensure superior platform health.

Requirements:

• Strong experience with microservices, messaging systems, databases, and infrastructure as code.
• Hands-on experience hosting large language or multimodal models using inference engines like vLLM, SGLang, or Modular.
• Familiarity with distributed inference serving frameworks such as llm-d, NVIDIA Dynamo, or Ray Serve.
• Understanding of GPU-level optimization and experience with interconnect technologies like NVlink, XGMI, or RoCE.
• Knowledge of common LLM architectures and optimization techniques (e.g., continuous batching, quantization).
• Expert-level proficiency in GoLang or Python and familiarity with gRPC.
• Proven experience shipping customer-facing software products and running critical services in a high-scale environment similar to DigitalOcean.
• Experience integrating and building with open-source software.

Benefits:

• Employee Assistance Program
• Local Employee Meetups
• Flexible time off policy
• Reimbursement for relevant conferences, training, and education
• Access to LinkedIn Learning's 10,000+ courses