Staff Data Engineer

Staff Data Engineer

Market America, a product brokerage and Internet marketing company that specializes in One-to-One Marketing, is seeking an experienced Staff Data Engineer for our IT team.

As a senior member of the Data Engineering team, you will play an important role in helping millions of customers on our SHOP.COM and Market America Worldwide multi-country and multi-language global eCommerce websites intelligently find what they want within different categories, merchant offers, products and taxonomy.

We have thousands of third party affiliates/feeds which go through ETL and data ingestion pipelines before ingesting into our search systems. We have multiple orchestration pipelines supporting various types of data for products, store offers, analytics, customer behavioral profiles, segments, logs and much more.

If you are passionate about data engineering in processing millions of data, ETL processes for products, stores, customers, analytics, this is a highly visible role that will provide you the opportunity to make a huge impact in our business and a difference to millions of customers worldwide. Data engineering team processes large amounts of data that we import and collect. The team works to enrich content, pricing integration, taxonomy assignments and algorithms, category classifier nodes and machine learning integration within the pipeline.

Key Responsibilities:

• Must have minimum of 10-12 years of hands-on development experience implementing batch and events driven applications using Java, Kafka, Spark, Scala, PySpark and Python

• Experience with Apache Kafka and Connectors, Java, Springboot in building event driven services, Python in building ML pipelines

• Develop data pipelines responsible for ingesting large amounts of different kinds of data from various sources
• Help evolve data architecture and work on Next Generation real time pipeline algorithms and architecture in addition to supporting and maintaining current pipelines and legacy systems
• Write code and develop worker node for business logic, ETL and orchestration processes
• Develop algorithms for better attribution rules and category classifiers
• Work with stakeholders throughout the organization to identify opportunities for leveraging company data to drive search, discovery, and recommendations.
• Work closely with architects, engineers, data analysts, data scientists, contractors/consultants and project managers in assessing project requirements, design, develop and support data ingestions and API services
• Work with Data Scientists in building feature engineering pipelines and integrating machine learning models during content enrichment process
• Able to influence on priorities working with various partners including engineers, project management office and leadership
• Mentor junior team members, define architecture, code review, hands-on development and deliver the work in sprint cycle
• Participate in design discussions with Architects and other team members for the design of new systems and re-engineering of components of existing systems
• Wear Architect hat when required to bring new ideas to the table, thought leadership and forward thinking
• Take holistic approach to building solutions by thinking big picture and overall solution
• Work on moving away from legacy systems into next generation architecture
• Take complete ownership from requirements, solution design, development, production launch and post launch production support. Participate in code reviews and regular on-call rotations
• Desire to apply best solution in the industry, apply correct design patterns during development and learn best practices and data engineering tools and technologies

Required Skills & Experience:

• BS or MS in Computer Science (or related field) with 10+ years of hands-on software development experience working in large-scale data processing pipelines
• Must have skills are Apache Spark, Scala and PySpark with 2-4 years of experience building production grade batch pipelines that handle large volumes of data.
• Must have at least 4+ years of experience in Java and API / Microservices
• Must have at least 2+ years of experience in Python
• 2+ years of experience in understanding and writing complex SQL and stored procedures for processing raw data, ETL, data validation, using databases such as SQL Server, Redis and other NoSQL DBs
• Knowledge of Big data technologies, Hadoop, HDFS
• Expertise with building events driven pipelines with Kafka, Java / Spark, Apache Flink
• Expertise with Amazon AWS stack such as EMR, EC2, S3
• Experience working with APIs to collect and ingest data as well build the APIs for business logic
• Experience working with setting up, maintaining, and debugging production systems and infrastructure
• Experience in building fault-tolerant and resilient system
• Experience in building worker nodes, knowledge of REST principles and data engineering design patterns
• In-depth knowledge of Java, Springboot, Spark, Scala, PySpark, Python, Orchestration tools, ESB, SQL, Stored procedures, Docker, RESTful web services, Kubernetes, CI/CD, Observability techniques, Kafka, Release processes, caching strategies, versioning, B&D, BitBucket / Git and AWS Cloud Eco-system, NoSQL Databases, Hazelcast
• Strong software development, architecture diagramming, problem-solving and debugging skills
• Phenomenal communication and influencing skills

Nice to Have:

• Exposure to Machine Learning (ML), LLM models, using AI during coding, build with AI
• Knowledge of Elastic APM, ELK stack and search technologies such as Elasticsearch / SOLR
• Nice to have some experience in workflow orchestration tools such as Air Flow or Apache NiFi

Market America offers competitive salary and generous benefits, including health, dental, vision, life, short and long-term disability insurance, a 401(k) retirement plan with company match, and an on-site health clinic. Qualified candidates should apply online. This position can work remotelyd based from either our Greensboro NC or Monterey CA offices. Sorry, we are NOT able to sponsor for this position. Market America is proud to be an equal opportunity employer.

Market America | SHOP.COM is changing the way people shop and changing the economic paradigm so anyone can become financially independent by creating their own economy and converting their spending into earning with the Shopping Annuity®.

Market America Worldwide | SHOP.COM is a global e-commerce and digital marketing company that specializes in one-to-one marketing and is the creator of the Shopping Annuity. Its mission is to provide a robust business system for entrepreneurs, while providing consumers a better way to shop. Headquartered in Greensboro, North Carolina, and with eight sites around the globe, including the U.S., Market America Worldwide was founded in 1992 by Founder, Chairman & CEO JR Ridinger. Through the company's primary, award-winning shopping website, SHOP.COM, consumers have access to millions of products, including Market America Worldwide exclusive brands and thousands of top retail brands. Further, SHOP.COM ranks 19th in Newsweek magazine's 2021 Best Online Shops, No. 52 in Digital Commerce 360's (formerly Internet Retailer) 2021 Top 1,000 Online Marketplaces, No. 79 in Digital Commerce 360's 2021 Top 1,000 Online Retailers and No. 11 in the 2021 Digital Commerce 360 Primary Merchandise Category Top 500. The company is also a two-time winner of the Better Business Bureau's Torch Award for Marketplace Ethics and was ranked No. 15 in The Business North Carolina Top 125 Private Companies for 2021. By combining Market America Worldwide's entrepreneurial business model with SHOP.COM's powerful comparative shopping engine, Cashback program, Hot Deals, ShopBuddy, Express Pay checkout, social shopping integration and countless other features, the company has become the ultimate online shopping destination.

For more information about Market America Worldwide: MarketAmerica.com

For more information on SHOP.COM, please visit: SHOP.COM