Data Engineer (remote, South Africa Only) - Remote Eligible

Data Engineer

Netstock's leading business intelligence products are underpinned by cutting edge expertise in data, optimization metrics and phenomenal software development. Our backers are expressly investing in R&D for product innovation, to drive revenue growth of 350% over the next 3 years. Our next innovation frontier is leveraging the 30+TB of customer data under our management, to develop disruptive products that none of our competitors can match. To help get us there, as well as ensure that we engineer data pipelines and workflows that can support the planned scaling to 130TB of data spread across 10,000 customer databases, we are looking for a data engineer of exceptional talent, vision, energy and ingenuity.

Responsibilities

• Technology Research and Testing: Investigate the most suitable technologies to build out a new data lake environment. Assess the pros and cons of different solutions and recommend the most efficient, cost-effective and scalable options.
• Infrastructure Liaising: Collaborate closely with the infrastructure team to communicate necessary infrastructure requirements for building and maintaining the data lake. Coordinate efforts and ensure all infrastructure needs are met.
• Data Lake Development: Design, construct, and deploy the data lake. Ensure it is built with scalability and security in mind to accommodate future data growth and needs.
• Data Pipeline Construction: Develop robust and efficient data pipelines for the ingestion, transformation, and loading of data into the data lake. This includes building pipelines from external sources (APIs) for internal analytics and benchmarking. Ensure the pipelines are optimized for real-time and batch processing, automating as many data processes and workflows as possible.
• Data Management: Manage both internal data and customer data. This involves cataloging, storing, retrieving, and maintaining the data in a manner that facilitates easy access and understanding. Data quality & integrity will be critical to ensure that every data process runs smoothly, is optimized and that the output can be trusted.
• Internal Data Provision & Reporting: Engineer the internal data to create useful and insightful reports, as well as consumable datasets for both technical and non-technical internal stakeholders. Ensure data integrity and accuracy of these datasets/reports.
• Customer Data Engineering: Work with customer data for benchmarking and future AI analysis. This includes data cleaning, standardization, transformation, and loading into the appropriate databases or platforms.
• Data Security and Compliance: Uphold data privacy and security standards, and ensure that all data handling practices are compliant with relevant laws & regulations, as well as ISO27001 standards.
• Collaboration with Data Analyst: Work closely with the data analyst to understand data needs and requirements. Make sure the data is structured and cleaned in a way that allows the data analyst to effectively conduct their analysis.
• Reporting to Head of Department: Regularly report progress, challenges, and successes to the head of the data department. Maintain open lines of communication to ensure alignment with broader strategic goals.
• Future-proofing: Stay up-to-date with the latest trends, technologies, and best practices in data engineering to ensure the data infrastructure is scalable and capable of accommodating future growth developments.

Circumstances

• Flexibility to travel within South Africa from time-to-time for team or company get-togethers.
• Netstock will provide the hardware necessary to perform this role (including UPS power).
• As this is a remote role, you'll need access to stable, secure, high-speed fixed-line internet connectivity (Netstock provides a subsidy towards internet subscription costs).

Working with us

Netstock was founded with a clear vision: To give the hungry up-and-comers the capability to level the playing field and compete with the industry giants. Working here means embracing that "challenger" mentality: We are smart, scrappy fighters, building our edge with the agility to move faster than the big guys — pioneering smarter ways to work and innovating new ways to deliver powerfully easy to use technologies for our customers.

About us

Netstock is the driving force accelerating the growth of organizations worldwide. Over the last 15 years, we've built out a regional presence that gives us deep insights into supply chain planning factors in each industry. We continue to enhance our supply chain planning solutions, making our predictive engine smarter, accelerating automation, and adding sophisticated new capabilities such as AI and machine learning.