Data Engineer
We seek an experienced Data Engineer proficient in Azure and Python, focused on building optimized data pipelines and ensuring data security.
Overview:
– We are seeking a highly skilled Data Engineer with 5 to 8 years of experience to join our Data Management team. The candidate must have extensive experience in Azure technologies and sophisticated Python skills. Experience with Oracle technologies would be an added advantage. This role involves crafting, building, and optimizing data pipelines, ensuring data security, and enabling analytical capabilities across the organization.
Responsibilities:
– Develop, test, and maintain high-quality software products by using ground breaking technologies and best programming practices.
– Design, develop, and maintain scalable and high-performance data pipelines using Azure Data Factory, Azure Synapse Analytics, and other Azure services.
– Develop and maintain ETL/ELT processes to ensure the quality, consistency, and accuracy of data.
– Implement data integration solutions across on-premise, hybrid, and cloud environments.
– Ensure the security, availability, and performance of enterprise data platforms.
– Work with relational (SQL Server, Azure SQL) and non-relational databases (Azure Cosmos DB, etc.).
– Build, test, and deploy data solutions using Azure DevOps, version control, and CI/CD pipelines.
– Develop and implement data governance policies and practices to ensure data integrity and security.
– Perform code reviews and ensure the quality of work from the other developers are as per the standards.
– Collaborate with multi-functional teams, including designers, developers, and quality assurance engineers to build, refine, and improve software products.
– Optimize and solve large datasets using Python and Azure cloud-native technologies.
– Build and maintain comprehensive user documentation.
Required Qualifications:
– Bachelor’s degree in computer science or a related field
– 5+ years of experience as a Data Engineer, focusing on Azure cloud services with a Solid understanding of Python/PySpark programming language.
– Strong hands-on experience with Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, and Azure Storage.
– Experience with Azure Databricks and PySpark is highly desirable
– Strong SQL skills, including experience with data modeling, sophisticated queries, and performance optimization.
– Expertise in Delta Tables and Delta live tables
– Familiarity with version control systems (e.g., Git) and CI/CD pipelines (Azure DevOps).
– Knowledge of Data Lake Architecture, Data Warehousing, and Data Modeling principles.
– Strong problem-solving, communication, and collaboration skills.
– In-depth understanding of the Python software development stacks, ecosystems, frameworks, and tools such as NumPy, SciPy, Pandas, sci-kit-learn and PyTorch.
– Writing scalable code using Python programming language, Multi-process architecture
– Integrating and running notebooks into Data Pipelines
– Able to work in the US without need for sponsorship now or in the future
Preferred Qualifications:
– Experience in ODI/OBIEE and PL/SQL would be a plus.
– Experience in implementing Medallion Architecture
– Understanding of Dimensional Modeling and Data Mesh Architecture is an added
– Experience with RESTful APIs, Data APIs, and event-driven architecture
– Familiarity with standard processes in data governance, lineage, security, and privacy.
– Strong analytical, communication, and teamwork skills.
– Microsoft Certified:
– Azure Solutions Architect Authority
– Microsoft Certified:
– Azure Fundamentals
Who You Are:
– A competent professional characterized by having a strong instinct to act, consistently pursuing self-development to enhance skills and knowledge. You possess a drive for achieving results, demonstrating a commitment to perfection in their endeavors. Tech-savvy professional adept at demonstrating technological tools to streamline processes and enhance productivity. You excel in collaborative efforts, recognizing the value of collaboration and optimally contributing to collective goals.