Staff Software Engineer Search Platform, Ingestion & Indexing

Overview Advanced Content Engineering (ACE) is seeking a Staff Software Engineer to serve as the technical anchor for the search platform’s ingestion and indexing systems. The platform processes millions of documents across legal, tax, and professional content corpora — parsing, chunking, enriching, embedding, and indexing them into a hybrid search engine that powers both human-facing search interfaces and autonomous AI agents. This role owns the design, implementation, and operational health of the document ingestion pipeline and search index management systems, from Kafka-based streaming infrastructure to Vespa application architecture. The expectation is full‑stack ownership and constant delivery to production, with a focus on architectural leadership. Responsibilities Ingestion Pipeline Architecture & Engineering Plan, design, develop, and own the end‑to‑end document ingestion pipeline — a Kafka‑based stream processing architecture that moves documents through parsing, chunking, enrichment (entity extraction, embedding generation, metadata enrichment), and indexing stages, with fault tolerance, version ordering, and at‑least‑once delivery guarantees. Architect and implement pluggable, configurable pipeline components that client teams can assemble into custom topologies via self‑service APIs, maintaining reliable, observable, and performant execution. Own the platform’s Protobuf‑based document schema and schema registry integration, establishing schema governance, enforcing backward compatibility, and ensuring reliable serialization. Design and implement dual‑flow ingestion: a high‑throughput batch path for full reindexing and a low‑latency incremental path for real‑time updates, with strong guarantees around version ordering and idempotency. Lead the migration of ingestion infrastructure from OpenSearch to Vespa, including custom Kafka feeders and application package architecture. Custom Model Operationalization Own the end‑to‑end lifecycle for custom models integrated into the ingestion pipeline — re‑ranking, embedding, and enrichment components, including inference serving, latency SLO management, batching, and scaling. Build and operate the model promotion pipeline (CI/CD workflow) from fine‑tuning through staging to production, with versioning, canary rollouts, and rollback mechanisms. Define and maintain integration contracts between models and downstream components, governing input/output schemas and governance processes. Instrument model serving for production observability: latency distributions, throughput, error rates, and quality signals like re‑ranking score distributions. Search Engine & Index Management Own the search engine layer end‑to‑end: design and operate Vespa (and transition from OpenSearch) index configurations, ranking profiles, schema definitions, and application package lifecycle management. Build and operate zero‑downtime index management: shadow indexing, blue/green promotion, and rolling reindex workflows. Implement and maintain the Component Registry and Index Registry, ensuring correctness, observability, and safe concurrent modification. Develop full‑reindex and incremental‑update orchestration logic, including change detection, document tracking, Kafka topic management, and DynamoDB‑backed state management. Agentic Search Infrastructure Design ingestion and indexing infrastructure with agentic retrieval patterns, explicit latency budgets, chunking and result compression strategies, and index boundary definitions. Build trace‑level observability into the retrieval stack to capture tool usage and order for deterministic diagnosis. Design session state and cache invalidation patterns for multi‑turn agentic search, reasoning on cache validity windows, session state scope, and stale‑context prevention. Evaluation & Search Quality Build and own the integration between the ingestion pipeline and the platform’s offline evaluation framework, supporting evaluation, grading, and ranking comparison. Instrument query and retrieval stack for online analytics: real‑time latency, throughput, query log collection, and support for A/B experiments. Partner with research scientists to evaluate new search components in isolation and drive promotion decisions. Reliability & Operational Ownership Take full operational responsibility: define SLOs, build CloudWatch dashboards and alarms, and participate in on‑call rotations. Improve CI/CD pipelines, deployment automation, and local development workflows to reduce friction. Instrument pipeline components with distributed tracing, structured logging, and rich metrics, establishing documentation standards. Design resilient fault tolerance mechanisms: dead‑letter queues, retry strategies, circuit breakers, and lag monitoring. Drive system‑level performance architecture: profiling throughput, indexing latency, identifying bottlenecks, and implementing optimizations. Technical Leadership Serve as the deepest technical authority on document pipelines and search engine internals, guiding architecture and resolving ambiguity. Lead cross‑team projects, set priorities, and adjust short‑term work while maintaining strategic focus. Mentor senior and mid‑level engineers, providing coaching and educational opportunities in distributed systems, stream processing, and AI‑assisted development. Collaborate with research scientists to integrate new chunking strategies, embedding models, and enrichment techniques responsibly. Deliver presentations to technical and non‑technical stakeholders, aligning technology plans with business objectives. Qualifications Required Experience Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field. 8+ years of software engineering experience, with progression to staff‑level or equivalent technical leadership. Deep expertise in distributed stream processing, designing high‑throughput, fault‑tolerant event‑driven pipelines using Kafka or equivalent. Production experience with Vespa, OpenSearch, or Elasticsearch (schema design, ranking profiles, application lifecycle). Mastery of Python, strong fundamentals in test strategy, performance architecture, and system design. Proficiency with AWS services (MSK, ECS, Lambda, DynamoDB, Step Functions, CloudWatch) and infrastructure‑as‑code (Terraform or CDK). Track record of end‑to‑end operational responsibility: defining SLOs, observability, on‑call, and systemic improvements. Comfort with AI‑assisted development tools, using them to accelerate high‑quality work. Evidence of establishing architectural principles and documentation standards that improve engineering quality. Preferred Experience Operationalizing ML models in production: inference serving, model promotion pipelines, canary rollouts, and observability. Familiarity with agentic retrieval patterns and latency budget management across hops. Experience with online search analytics, A/B or interleaved ranking experiments, and query log analysis. Experience with embedding pipelines, vector indexing, and hybrid retrieval architectures. Familiarity with Protobuf schema design and schema registry governance. Experience building self‑service or multi‑tenant platform infrastructure with high reliability. Background in AI ethics frameworks and responsible deployment of ML components. What Success Looks Like First 90 Days Develop a thorough understanding of the current ingestion and indexing architecture, technical debt, reliability gaps, and Vespa roadmap. Establish working relationships with the search platform team, research labs, and client teams. Deliver at least one meaningful improvement to pipeline reliability, observability, or delivery automation. First Year Lead architectural design and delivery of a major phase of the Vespa migration, including zero‑downtime index promotion. Establish robust SLO coverage and observability across ingestion components, with on‑call playbooks and documented decisions. Deliver a production‑ready custom model operationalization framework. Become the recognized technical authority for ingestion and indexing, providing architectural direction and influencing platform strategy. Benefits Hybrid work model with flexible in‑office days. Comprehensive benefits package: health, dental, vision, disability, life insurance, retirement savings, tuition reimbursement, and wellness programs. Paid vacation, sick time, mental health days, parental leave, and other paid time off. Employee assistance programs, Headspace app, fitness reimbursement, and more. Social impact opportunities and volunteer days off. Equal Employment Opportunity Thomson Reuters is proud to be an Equal Employment Opportunity Employer providing a drug‑free workplace. We make reasonable accommodations for applicants with disabilities and are committed to a diverse and inclusive workforce. #J-18808-Ljbffr