Comprehensive Aggregated Job Description

Primary Languages: - Python - Primary language for ML development, data processing, and automation - **C** - High-performance computing, GPU kernels, and systems programming - **Go (Golang)** - Kubernetes controllers, infrastructure tooling, and distributed systems - **Rust** - Systems programming and performance-critical components (mentioned as alternative to C)

Supporting Languages: - JavaScript/TypeScript - Front-end development, Node.js, React - HTML/CSS - Web interfaces and dashboards - Bash - Scripting and automation - Assembly - Low-level optimization and hardware-specific code

Deep Learning Frameworks: - PyTorch - Primary framework for model development, training, and inference - TensorFlow - Alternative framework for model development and deployment - JAX - Research and high-performance ML computations - MIOpen - AMD’s deep learning primitives library

Compilation and Optimization Tools: - TorchScript - PyTorch model compilation - TorchDynamo - PyTorch dynamic compilation - MLIR/LLVM - Compiler infrastructure and optimization - TVM - Tensor expression compilation - TensorFlow MLIR - TensorFlow compilation pipeline

GPU Programming: - CUDA - NVIDIA GPU programming (most common) - OpenCL - Cross-platform GPU programming - HIP - AMD GPU programming (CUDA-like API) - ROCm - AMD’s open software platform for GPU computing - Triton - GPU kernel development and optimization

GPU Libraries: - cuDNN - NVIDIA’s deep neural network library - cuBLAS - NVIDIA’s BLAS library - CUTLASS - CUDA Templates for Linear Algebra Subroutines - TensorRT - NVIDIA’s inference optimization library

High-Performance Inference: - vLLM - High-throughput LLM serving - SGLang - Structured generation language for LLMs - TRT-LLM (TensorRT-LLM) - Optimized LLM inference - TensorRT - NVIDIA inference optimization - ONNX Runtime - Cross-platform inference engine

Container and Orchestration: - Kubernetes - Container orchestration (universally required) - Docker - Containerization - OpenShift - Enterprise Kubernetes platform (RedHat) - Singularity - HPC container platform

CI/CD and Build Systems: - Jenkins - Continuous integration (with Groovy scripting) - GitHub Actions - CI/CD workflows - GitLab Pipelines - CI/CD automation - Azure DevOps - Microsoft’s DevOps platform - CMake - Build system configuration - Bazel - Build and test tool - Make - Traditional build automation

Infrastructure as Code: - Crossplane - Cloud-native infrastructure automation - Terraform - Infrastructure provisioning (implied) - Ansible - Configuration management (implied)

Major Cloud Providers: - AWS (Amazon Web Services) - Cloud infrastructure - GCP (Google Cloud Platform) - Cloud infrastructure - Azure (Microsoft Azure) - Cloud infrastructure

Distributed ML: - PyTorch DDP - Distributed Data Parallel training - DeepSpeed - Deep learning optimization library - Ray - Distributed computing framework - Spark - Big data processing

Kubernetes ML Tools: - Kubeflow - ML toolkit for Kubernetes - Kueue - Job queue management for Kubernetes - Custom Kubernetes Operators - Application-specific orchestration

Performance Profiling: - Nsight - NVIDIA performance analysis - nvprof - NVIDIA CUDA profiler - CUPTI - CUDA Profiling Tools Interface - VTune - Intel performance profiler

System Monitoring: - Prometheus - Metrics collection - Grafana - Metrics visualization and dashboards

Data Processing: - ETL Pipelines - Extract, Transform, Load workflows - Data Quality Tools - Quality assurance and validation - Data Filtering and Deduplication - Data cleaning - Distributed Data Processing - Spark, Ray, etc.

Agent Development: - LangChain - LLM application framework - CrewAI - Multi-agent orchestration framework

Source Control: - Git - Version control (primary) - Perforce - Enterprise version control

Project Management: - Jira - Issue tracking and project management - GitLab - Complete DevOps platform

Supported Platforms: - Ubuntu - Linux distribution - RedHat/RHEL - Enterprise Linux - Windows - Microsoft OS - QNX - Real-time operating system

HPC and Batch Systems: - Slurm - Workload manager for Linux clusters - Kubernetes Job Scheduling - Container-based job management

Inference Optimization: - Optimize large language models (LLMs) for inference performance - Implement quantization techniques (post-training and quantization-aware training) - Apply pruning and distillation methods - Optimize transformer architectures for inference efficiency - Benchmark and evaluate inference performance across different hardware - Reduce memory use and compute cost with mixed precision, KV-cache handling, and speculative decoding

Training Optimization: - Design and implement distributed training systems - Optimize data loading and preprocessing pipelines - Implement fault tolerance and checkpointing for long-running training jobs - Monitor and optimize training performance - Work with distributed training frameworks (PyTorch DDP, DeepSpeed)

Model Compilation: - Work with TorchScript, TorchDynamo, and PyTorch compilation tools - Optimize ML models for inference on specialized hardware (Cerebras, etc.) - Research and implement novel inference optimization techniques

Kubernetes and Orchestration: - Design and implement Kubernetes-native systems to orchestrate GPU workloads - Develop custom Kubernetes operators and CRDs - Deploy, configure, and maintain highly available Kubernetes clusters - Optimize cluster performance, capacity planning, and scaling - Implement security best practices (RBAC, Pod Security Policies, network policies) - Contribute to job scheduling mechanisms (gang scheduling, fair sharing, opportunistic compute)

GPU Orchestration: - Optimize GPU utilization and throughput across multi-node clusters - Build and maintain Kubernetes-based orchestration for inference workloads - Design and implement distributed vLLM inference systems - Work on model parallelism and pipeline parallelism - Optimize multi-GPU and multi-node serving performance

Infrastructure Automation: - Develop and maintain infrastructure-as-code (IaC) using Crossplane - Write scripts and tools (Python, Bash, Go) to automate routine tasks - Build scalable automation for build, test, integration, and release processes - Configure, maintain, and build upon deployments of industry-standard tools

Model Deployment: - Build and maintain inference serving infrastructure - Design and implement scalable deployment systems - Deploy models to production environments - Automate model deployment and scaling - Integrate MLOps tools with enterprise platforms (OpenShift, etc.)

CI/CD Pipelines: - Design and implement CI/CD pipelines for model deployment - Collaborate with development teams to optimize CI/CD pipelines - Use GitHub Actions, GitLab pipelines, or Azure DevOps for automation - Build automation and tooling for ML workflows

Monitoring and Observability: - Set up monitoring, logging, and alerting systems (Prometheus, Grafana) - Implement automated recovery procedures - Monitor and maintain production ML systems - Work on monitoring, observability, and alerting for LLM serving - Troubleshoot incidents to minimize downtime

Performance Analysis: - Specify test cases derived from Deep Learning workloads - Determine performance theory through analytical models - Track and report on kernel performance throughout development lifecycle - Identify performance regressions and optimization opportunities - Profile real workloads and remove bottlenecks - Analyze architecture performance and energy efficiency

GPU Kernel Development: - Write highly tuned compute kernels in C++ CUDA - Perform core deep learning operations (matrix multiplies, convolutions, normalizations) - Design, implement, and test GPU kernels for tensor operations - Optimize kernel performance for current and future-generation GPUs

Benchmarking: - Build repeatable tests that model production traffic - Track and report on vLLM, SGLang, TRT-LLM, and future runtimes - Benchmark, analyze, and optimize performance on single and multi-GPU systems

Algorithm Development: - Research and develop new AI/ML techniques and algorithms - Implement and evaluate research ideas - Work on model training, optimization, and deployment - Publish research findings and contribute to open-source projects - Stay up-to-date with the latest research in AI/ML

Architecture Design: - Craft high-performance, energy-efficient system and processor architectures - Prototype key deep learning and data analytics algorithms - Analyze trade-offs in performance, cost, and power - Collaborate across teams to guide the direction of machine learning

Data Pipeline Development: - Build and maintain data pipelines for training large language models - Design and implement data processing and preprocessing systems - Work on data quality, filtering, and deduplication - Optimize data loading and processing for distributed training - Build data processing and ETL pipelines from scratch

Forward Deployment: - Work directly with customers to understand their AI needs - Deploy and customize AI solutions for customer use cases - Provide technical support and guidance to customers - Gather customer feedback and communicate requirements to product teams - Build custom integrations and solutions - Own projects end-to-end from first conversation to production rollout

Developer Advocacy: - Create technical content (blog posts, tutorials, demos) about AI technologies - Give talks and presentations at conferences and meetups - Engage with the developer community on forums, social media, and events - Help developers troubleshoot and solve problems with AI deployments

Product Management: - Act as a product manager in the field - Capture qualitative and quantitative feedback - Synthesize themes and translate them into clear product requirements - Influence product roadmap and direction

Code Quality: - Follow general software engineering best practices - Support regression testing and CI/CD flows - Deliver high-quality code and documentation - Implement best practices for open-source software development - Design for safety, observability, and robustness

Testing and Quality Assurance: - Design and develop software for testing and analysis - Build unit and integration test frameworks - Implement automated testing in CI/CD pipelines

Front-End Development: - Develop front-end solutions using HTML, CSS, JavaScript, and related web technologies - Build user interfaces and dashboards - Work with React, NodeJS, or similar frameworks

Academic Credentials: - Bachelor’s Degree - Computer Science, Computer Engineering, Electrical Engineering, Applied Math, or related field (minimum for most positions) - Master’s Degree - Preferred for many senior roles, required for some research positions - PhD - Required for research roles, preferred for senior research and architecture positions - Equivalent Experience - Industry experience can substitute for formal education

Specialized Fields: - Computational Chemistry, Bio Physics, Bioinformatics (for life sciences roles) - Machine Learning, Artificial Intelligence - Computer Architecture - High-Performance Computing

Years of Experience (by role level): - Entry Level / New College Grad - 0-1 years - Junior / Associate - 2-3 years - Mid-Level - 3-5 years - Senior - 5-7 years - Principal / Lead - 7+ years (8+ for some lead positions)

Relevant Experience Areas: - ML engineering or systems engineering - MLOps or DevOps - Infrastructure engineering - Research in deep learning and AI - GPU computing and parallel programming - Distributed systems and cloud infrastructure - Customer-facing technical roles (for forward deployed positions)

Programming Proficiency: - Strong C++ programming - Required for performance-critical roles - Strong Python programming - Required for all ML roles - Go/Golang - Required for infrastructure and Kubernetes roles - JavaScript/TypeScript - For front-end and full-stack roles - Assembly programming - For low-level optimization roles

ML/AI Expertise: - Strong experience with PyTorch or TensorFlow - Deep understanding of transformer architecture and inference engine internals - Knowledge of model optimization techniques (quantization, pruning, distillation, etc.) - Experience with model serving frameworks (vLLM, TensorRT-LLM, ONNX Runtime, etc.) - Strong research background in deep learning and AI (for research roles)

GPU and Parallel Computing: - Experience with CUDA, OpenCL, HIP, or ROCm - Understanding of GPU architectures and low-level optimization techniques - Knowledge of memory hierarchy, instruction scheduling, and performance tradeoffs - Experience with performance profiling tools (Nsight, nvprof, CUPTI, VTune)

Systems and Infrastructure: - Strong experience with Kubernetes and container orchestration - Experience with distributed systems and cloud infrastructure - Knowledge of CI/CD systems (Jenkins, GitHub Actions, GitLab, Azure DevOps) - Experience with infrastructure-as-code tools - Understanding of cloud platforms (AWS, GCP, Azure)

Data Engineering: - Experience with data engineering and ETL pipelines - Proficiency in data processing libraries - Experience with distributed data processing (Spark, Ray, etc.) - Knowledge of data storage systems and formats

Communication: - Excellent written and verbal communication skills - Ability to translate deep technical topics for diverse audiences - Strong presentation skills (for developer advocate and customer-facing roles) - Strong community engagement and networking skills

Problem Solving: - Strong problem-solving skills - Ability to navigate ambiguity - Pragmatic technical decision-making - Systems thinking and ability to design for safety and robustness

Collaboration: - Ability to work effectively in fast-paced, collaborative environments - Experience working with cross-functional teams - Strong collaboration skills - Ability to mentor junior engineers (for senior/principal roles)

Leadership: - Experience leading or managing teams (for lead/principal roles) - Ability to make key technical decisions and architecture choices - Entrepreneurial mindset (for startup roles) - Ability to work independently and manage multiple projects

Learning: - Ability to learn and work effectively in fast-paced environments - Desire and ability to quickly acquire broad high-level intuition of AI systems - Stay up-to-date with latest research and technologies

Computer Architecture: - Solid understanding of computer architecture - Experience with analytical performance modeling - Knowledge of high-performance, power-efficient designs - Experience with cycle-accurate HW simulators

Numerical Methods: - Numerical methods and linear algebra - Linear algebra operations (matrix multiplies, convolutions) - BLAS operations

Distributed Systems: - Experience building distributed systems - Knowledge of model parallelism and pipeline parallelism - Experience with distributed training and inference - Understanding of job scheduling mechanisms

Open Source: - Contributions to open-source projects (preferred) - Experience working with open-source communities - Track record of blog posts, conference talks, or open-source projects

Publications: - Publications in top-tier ML conferences (NeurIPS, ICML, ICLR) - Preferred for research roles

Life Sciences: - BSc, MSc, PhD in Computational Chemistry, Bio Physics, Bioinformatics, or related field - Experience with life sciences software development

Enterprise Platforms: - Experience with Red Hat platforms (OpenShift, RHEL) - For RedHat roles - Knowledge of enterprise deployment patterns

Agent Systems: - Experience building applications with LLMs or foundation models - Familiarity with AI agent frameworks (LangChain, CrewAI) - Knowledge of planning algorithms or program synthesis using LLMs