Full Stack Data Engineer

Job Description:

We need Elasticsearch, Gen AI, LLMs/AI engineering skills, and strong API engineering and test driven development skills with python. Kubernetes based deployments desirable. Cloud skills are a must. Knowledge of shipping production grade code is essential.

KEY RESPONSIBILITIES

· Should be self-motivated and with a go-getter attitude, working under ambiguity and with at times incomplete requirements.

· Strong abilities to deliver innovative new solutions as a full stack engineer working on data science and data engineering POCs and projects

· Good working knowledge of Azure Cloud or AWS and various aspects of these platforms such as IAM, storage, compute, services and building applications on them.

· Knowledge of SQL and the ability to work with a SQL database in the backend, such as PostgreSQL.

· Knowledge of Apache Spark, Apache Dask and similar distributed data processing frameworks and the Databricks platform

·Conversant with the development and deployment of Python web services and APIs for different architectures, such as synchronous and asynchronous REST API calling, event-driven ML product architectures and batch processing architectures for reporting and BI use cases

·Knowledge on JavaScript/TypeScript and the ability to develop simple and functional front-ends with frameworks such as React or Vue

·Conversant with one statically typed language and the development of applications in it. C# .NET, or Java knowledge desirable but not necessary

·Some knowledge of Elastic stack (Elastic, Logstash, Kibana), Databricks, vector databases such as Qdrant, and caches such as Redis preferred

·Ability to understand, develop and maintain code bases with performant, well tested and readable code, good test coverage and adherence to code standards such as PEP8 and .NET coding conventions

· Conversant with Git, Docker, CI/CD and the deployment of applications on Azure

· Logging and monitoring using tools such as Azure Log Analytics, Monitor, Prometheus

· Familiar with the sequential development, testing and promotion of code through different environments, leading up to production release.

· Desirable if the candidate is conversant with parallel processing and concurrency paradigms in Python and .NET

· Effectively collaborates with different team members such as data scientists, engineers, and architects to build and release data and AI applications.

· Clear verbal and written communication skills in English, and the ability to interact well with a globally distributed and diverse data science and engineering team.