Software Engineer (Data Engineering)
To see similar active jobs please follow this link: Remote Development jobs
What we are looking for
Equilibrium was founded with a vision for building a company where data-centric decision-making drives our operations. Our reliance on closed-loop, ML-infused, autonomous workflow decision-making puts data and data management at the foundation of everything we build and operate. We are looking for talented and passionate software engineering professionals to design, lay and support the technical foundation of our company – data – and thus drive our innovative, collaborative, and agile delivery culture.
What you will do
Support the design, development, implementation and maintenance of our data engineering tools, workflows, processes and platforms that underpin our data engineering efforts
Design and build infrastructure for optimal extraction, transformation and loading of data from various data sources (e.g., AWS tooling, Databricks, DBT, Snowflake, Temporal, and SQL technologies)
Identify, design, and implement internal process improvements including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes
Design and engineer performant, high-reliability systems to handle petabytes of data
Architecting, mentoring, implementing, and supporting a data ingestion platform for cross company use and enablement
Building and designing the long term vision for data ingestion at the company to enables cross team collaboration and end to end data ownership
Assemble large, complex sets of data that meet non-functional and functional business requirements and build large scaled data pipelines
Design and build analytical tools to accelerate and automate the implementation of new and changing data pipelines
Nurture, improve and maintain our data quality, including data accuracy, availability performance, and ease of access / understanding of the data in our data engineering ecosystem
Ensure data pipelines provide the intended actionable insights into key business performance metrics including operational efficiency
Design and implement scalable data validations per business and domain requirements.
Support a culture of continuous data quality improvement
Serve as a collaborative subject matter expert on our data and accelerate our understanding and use of our data
Work with stakeholders including executive, product, data and design teams to support their data engineering needs while assisting with data quality and comprehension
The minimum qualifications you’ll need
Passion for clean energy and fighting climate change
An advanced degree in computer science, data science, machine learning, artificial intelligence, operations research, engineering, or related quantitative discipline
8+ years of experience developing a globally distributed data services platform
8+ years coding experience (Python, Go, Java or other applicable languages)
Expertise in data analysis and ETL/ELT
Experience developing and operating large-scale data pipelines demonstrating knowledge with technologies, such as Airflow, Luigi, Spark, Kafka, ElasticSearch, Cassandra, RDBMS and SQL
Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases
Experience building and optimizing big data data pipelines, architectures and data sets
Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement
Strong analytic skills related to working with unstructured datasets
Build processes supporting data transformation, data structures, metadata, dependency and workload management
A successful history of manipulating, processing and extracting value from large disconnected datasets
A collaboration-first mentality, with a willingness to teach as well as learn from others
Nice to have additional skills
Familiarity with time series data transformation and ingestion
Strong understanding of data science concepts
Expertise in distributed database systems, data lakes and data meshes
About the job
Software Engineer (Data Engineering)
To see similar active jobs please follow this link: Remote Development jobs
What we are looking for
Equilibrium was founded with a vision for building a company where data-centric decision-making drives our operations. Our reliance on closed-loop, ML-infused, autonomous workflow decision-making puts data and data management at the foundation of everything we build and operate. We are looking for talented and passionate software engineering professionals to design, lay and support the technical foundation of our company – data – and thus drive our innovative, collaborative, and agile delivery culture.
What you will do
Support the design, development, implementation and maintenance of our data engineering tools, workflows, processes and platforms that underpin our data engineering efforts
Design and build infrastructure for optimal extraction, transformation and loading of data from various data sources (e.g., AWS tooling, Databricks, DBT, Snowflake, Temporal, and SQL technologies)
Identify, design, and implement internal process improvements including re-designing infrastructure for greater scalability, optimizing data delivery, and automating manual processes
Design and engineer performant, high-reliability systems to handle petabytes of data
Architecting, mentoring, implementing, and supporting a data ingestion platform for cross company use and enablement
Building and designing the long term vision for data ingestion at the company to enables cross team collaboration and end to end data ownership
Assemble large, complex sets of data that meet non-functional and functional business requirements and build large scaled data pipelines
Design and build analytical tools to accelerate and automate the implementation of new and changing data pipelines
Nurture, improve and maintain our data quality, including data accuracy, availability performance, and ease of access / understanding of the data in our data engineering ecosystem
Ensure data pipelines provide the intended actionable insights into key business performance metrics including operational efficiency
Design and implement scalable data validations per business and domain requirements.
Support a culture of continuous data quality improvement
Serve as a collaborative subject matter expert on our data and accelerate our understanding and use of our data
Work with stakeholders including executive, product, data and design teams to support their data engineering needs while assisting with data quality and comprehension
The minimum qualifications you’ll need
Passion for clean energy and fighting climate change
An advanced degree in computer science, data science, machine learning, artificial intelligence, operations research, engineering, or related quantitative discipline
8+ years of experience developing a globally distributed data services platform
8+ years coding experience (Python, Go, Java or other applicable languages)
Expertise in data analysis and ETL/ELT
Experience developing and operating large-scale data pipelines demonstrating knowledge with technologies, such as Airflow, Luigi, Spark, Kafka, ElasticSearch, Cassandra, RDBMS and SQL
Advanced working SQL knowledge and experience working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases
Experience building and optimizing big data data pipelines, architectures and data sets
Experience performing root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement
Strong analytic skills related to working with unstructured datasets
Build processes supporting data transformation, data structures, metadata, dependency and workload management
A successful history of manipulating, processing and extracting value from large disconnected datasets
A collaboration-first mentality, with a willingness to teach as well as learn from others
Nice to have additional skills
Familiarity with time series data transformation and ingestion
Strong understanding of data science concepts
Expertise in distributed database systems, data lakes and data meshes