Role Summary  

Design, build, and run core services and reusable components for the open-architecture data platform, enabling scalable ingestion, metadata-driven processing, governance, and reliable multi-engine data consumption. Partner with Platform Engineering and Data Engineering teams to provide secure, observable, production-grade capabilities that support Core Reference, IBOR, and Stamford data initiatives.  

Key Responsibilities  

• Design and build backend services and shared libraries that provide platform capabilities, including:  

• Streaming & batch ingestion frameworks  

• Metadata-driven pipelines  

• API and consumption layer integrations  

• Transformation engine and MDM engine  

• Engineer reusable platform components aligned with open formats (Iceberg, Parquet), catalog systems (Polaris, Horizon, Unity Catalog), and query engines (Spark, Trino)  

• Implement platform security controls including role-based access control, authentication, and integration with enterprise entitlements  

• Build and maintain platform observability (logging, metrics, traces), alerting, and operational dashboards in support of SLAs  

• Collaborate with architects to define scalable, extensible designs and ensure adherence to open standards  

• Drive engineering excellence through code reviews, automated testing, CI/CD, and production readiness practices  

• Ensure platform services meet enterprise security and compliance requirements (SCC, governance)  

• Participate in on-call / major incident management and troubleshoot production issues to maintain platform stability  

Qualifications  

• Strong programming skills in Python and Java; knowledge of Rust is a bonus  

• Strong programming skills in Spark, with hands-on knowledge of optimization and debugging  

• Hands-on experience with Snowflake and Databricks as data platforms  

• Strong understanding of SQL optimization, execution, and large-scale analytics  

• Deep understanding and working knowledge of open table formats such as Apache Iceberg are preferred  

• Deep understanding of distributed computing and hyperscale  

• Experience building production-grade services in cloud environments; AWS,GCP and/or Azure is preferred  

• Familiarity with data catalogs (any of Polaris, Horizon, Unity) and metadata management  

• Familiarity with streaming pipelines and Kafka is preferred  

• Understanding of data modeling and data product design  

• Experience with CI/CD, containers, and Kubernetes  

• Financial/data platform background preferred but not required