Junior Data Engineer - Databricks

Data Engineer

Our client is a leading health solutions organization with over 300,000 dedicated employees nationwide. As the Data Engineer on the Lumina team, you will own the analytics and telemetry foundation for our platform. You'll design and operate the data pipelines that power internal and external dashboards for product and engineering partners and end users. You will instrument and collect telemetry with OpenTelemetry, model and serve trustworthy metrics, and manage the data workflows supporting our vector database that enables Lumina's GenAI/RAG experiences. This role is ideal for an engineer who thrives at the intersection of data platforms, observability, and product outcomes, and who is hands on in Databricks and modern data/analytics tooling.

Key responsibilities include:

• Own Lumina analytics end to end: define metric sources of truth, build semantic layers, and deliver reliable, documented datasets for self service and curated dashboards.
• Design and operate data pipelines in Databricks: implement ELT/ETL with PySpark/SQL, Delta Lake, and Databricks Jobs (orchestrations), including testing, monitoring, and cost/performance optimization.
• Telemetry instrumentation & observability: design event schemas, implement tracing/metrics/logs using OpenTelemetry, and build health KPIs (latency, throughput, error budgets, SLOs) for Lumina services and data products.
• Dashboards for internal & external audiences: build, publish, and govern high quality Databricks dashboards for leadership, product/engineering teams, and end users; manage refresh SLAs and access controls.
• Vector database support: build and maintain ingestion/indexing pipelines for embeddings and metadata; implement data quality checks, drift monitoring, and retention policies for the vector store (MongoDB).
• Data quality & governance: incorporate validation (unit/integration tests), documentation, lineage, and incident playbooks; partner with security to ensure PII/PHI handling, privacy, and compliance controls.
• Collaboration & product mindset: partner with Product, ML/Platform Engineering, and Support to define analytics requirements, operationalize new telemetry, and iterate dashboards based on feedback and adoption metrics.
• Operational excellence: establish alerting and runbooks for pipelines; drive root cause analysis and continuous improvement for stability, cost, and developer efficiency.

Compensation: $30/hr to $40/hr. Exact compensation may vary based on several factors, including skills, experience, and education. Benefit packages for this role will start on the 1st day of employment and include medical, dental, and vision insurance, as well as HSA, FSA, and DCFSA account options, and 401K retirement account access with employer matching. Employees in this role are also entitled to paid sick leave and/or other paid time off as provided by applicable law.

We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.