Data Engineer
To see similar active jobs please follow this link: Remote Development jobs
Job Description:
As a Data Engineer at Everstream Analytics, you will play a critical role in building and maintaining our data infrastructure. You will work with a team of talented engineers to design, develop, and optimize data pipelines and data products that support our multi-tenant cloud-native data platform, leveraging various AWS services such as Lambda, EMR, S3, Glue and Redshift as well as helping drive our future toolset. Your expertise in distributed system design, data warehousing, data lakes, and ETL/Orchestration is essential in ensuring the scalability, reliability, and efficiency of our data infrastructure.
Key Responsibilities:
Design, implement, and maintain data pipelines that handle large volumes of data from various sources, ensuring data quality, integrity, and availability.
AWS Expertise: Utilize AWS services like Lambda, EMR, S3, Glue, and others to create scalable and cost-effective data solutions.
Relational Database Experience: Utilize PostgreSQL on RDS or similar database technologies, where applicable.
Stream Processing: Experience with Apache Kafka, Apache Spark or similar for real-time data processing and stream analytics.
Python Development: Primarily use Python for data engineering tasks, data transformation, and ETL processes.
Data Warehousing: Implement and manage data warehousing and/or data lake solutions for efficient data storage and retrieval to support engineering, data science, applications, and groups across our organization.
Collaboration: Work closely with Product Management, Data Science, and the leadership team to understand data requirements and deliver data solutions that meet business needs.
Monitoring and Optimization: Continuously monitor the performance of data pipelines to optimize scalability and efficiency.
Documentation: Maintain comprehensive documentation for data engineering processes, ensuring knowledge transfer within the team.
Leadership: Lead by example within the data engineering team, taking pride in your team’s deliverables, and performing as technical lead for a scrum team or on various projects, where applicable.
Qualifications:
Proven experience in designing and building multi-tenant cloud-native data platforms in a SaaS or PaaS environment.
Strong experience with Cloud Data Warehouses such as AWS Redshift, Snowflake, BigQuery, Databricks.
Extensive experience with relational database technologies in a production environment, specifically PosgreSQL.
Strong expertise in AWS services and ETL/Orchestration (Glue, Spark, Airflow, Apache Seatunnel).
Proficiency in distributed system design, data warehousing, data lakes, and stream processing using Spark or similar.
Strong programming skills in Python.
Excellent problem-solving and troubleshooting skills.
Ability to work collaboratively with cross-functional teams and convey complex technical concepts to non-technical stakeholders.
Bachelor's or Master's degree in Computer Science, Data Engineering, related field, or equivalent experience.
100% Remote Position
Applicants must be currently authorized to work in the United States on a full-time basis.
#LI-AB1
Data Engineer
To see similar active jobs please follow this link: Remote Development jobs
Job Description:
As a Data Engineer at Everstream Analytics, you will play a critical role in building and maintaining our data infrastructure. You will work with a team of talented engineers to design, develop, and optimize data pipelines and data products that support our multi-tenant cloud-native data platform, leveraging various AWS services such as Lambda, EMR, S3, Glue and Redshift as well as helping drive our future toolset. Your expertise in distributed system design, data warehousing, data lakes, and ETL/Orchestration is essential in ensuring the scalability, reliability, and efficiency of our data infrastructure.
Key Responsibilities:
Design, implement, and maintain data pipelines that handle large volumes of data from various sources, ensuring data quality, integrity, and availability.
AWS Expertise: Utilize AWS services like Lambda, EMR, S3, Glue, and others to create scalable and cost-effective data solutions.
Relational Database Experience: Utilize PostgreSQL on RDS or similar database technologies, where applicable.
Stream Processing: Experience with Apache Kafka, Apache Spark or similar for real-time data processing and stream analytics.
Python Development: Primarily use Python for data engineering tasks, data transformation, and ETL processes.
Data Warehousing: Implement and manage data warehousing and/or data lake solutions for efficient data storage and retrieval to support engineering, data science, applications, and groups across our organization.
Collaboration: Work closely with Product Management, Data Science, and the leadership team to understand data requirements and deliver data solutions that meet business needs.
Monitoring and Optimization: Continuously monitor the performance of data pipelines to optimize scalability and efficiency.
Documentation: Maintain comprehensive documentation for data engineering processes, ensuring knowledge transfer within the team.
Leadership: Lead by example within the data engineering team, taking pride in your team’s deliverables, and performing as technical lead for a scrum team or on various projects, where applicable.
Qualifications:
Proven experience in designing and building multi-tenant cloud-native data platforms in a SaaS or PaaS environment.
Strong experience with Cloud Data Warehouses such as AWS Redshift, Snowflake, BigQuery, Databricks.
Extensive experience with relational database technologies in a production environment, specifically PosgreSQL.
Strong expertise in AWS services and ETL/Orchestration (Glue, Spark, Airflow, Apache Seatunnel).
Proficiency in distributed system design, data warehousing, data lakes, and stream processing using Spark or similar.
Strong programming skills in Python.
Excellent problem-solving and troubleshooting skills.
Ability to work collaboratively with cross-functional teams and convey complex technical concepts to non-technical stakeholders.
Bachelor's or Master's degree in Computer Science, Data Engineering, related field, or equivalent experience.
100% Remote Position
Applicants must be currently authorized to work in the United States on a full-time basis.
#LI-AB1