Senior Data Engineer
To see similar active jobs please follow this link: Remote Development jobs
POS-4028
We are seeking a Senior Data Engineer to join our team. In this role, you will work with various external data sources, design and maintain data pipelines, and ensure the efficiency and security of our data processes. Your contributions will help HubSpot leverage data-driven insights for effective decision-making.
This role resides within our Employee Technology group which comprises Business Systems Analysts, System Engineers, Analytic Engineers and administrators responsible for HubSpot's employee-facing application infrastructure.
In this role you will:
Data Acquisition and Ingestion
Acquire data from multiple sources using pre-built connectors and custom solutions.
Develop data ingestion pipelines to effectively manage internal and external API interactions, addressing challenges such as rate limiting, pagination, and retry mechanisms.
Process and securely ingest large datasets, ensuring encrypted data transfer and runtime credential safety.
Implement Snowpipe for data ingestion into Snowflake and utilize Snowflake’s external tables for accessing S3 data.
AWS Infrastructure and File Management
Manage and optimize data workflows within the AWS ecosystem, with a focus on S3 operations.
Manipulate S3 files upon arrival, archive older files to Glacier, and do light cleansing/alteration of files, as needed, for various workflows.
Automate AWS processes using tools like AWS Lambda, Step Functions, or custom scripts to improve efficiency and reliability.
Data Migrations and Utility Development
Develop scripts for data migrations between Snowflake instances and automate tasks using Looker’s APIs.
Create utility scripts for business needs, such as disabling inactive users, purging broken/unused Looker content, and simplifying otherwise burdensome/manual processes.
Data Modeling and Development
Build and maintain data models using dbt, following best SQL practices and data warehousing principles to ensure schema design and data integrity.
Develop data pipelines in a DAG structure, ensuring clear dependencies and performing data validation and testing.
Implement CI/CD workflows with GitHub actions/hooks for automated testing and deployment.
We are looking for people who have:
Proven experience in developing custom data ingestion pipelines and integrating with APIs, effectively managing complexities such as rate limiting and pagination.
Proficient in automating workflows using AWS Lambda, Step Functions, or comparable tools.
Extensive knowledge of databases (especially Snowflake.) Experience with Snowflake technologies such as Snowpipe, Snowpark, Data Sharing, Cloning etc., and Snowpipe usage.
Strong background in SQL for data warehouse transformations. Experience with dbt is a plus.
Solid understanding of data warehousing principles, DAG structures, and comprehensive data validation/testing frameworks.
Skilled in Python for creating utility scripts and automated workflows.
Experience with data manipulation libraries such as Pandas, PySpark, or similar technologies.
Familiarity with streaming data solutions, including Kafka, AWS Kinesis, RabbitMQ, etc
Cash compensation range: 157600-236400 USD Annually This resource will help guide how we recommend thinking about the range you see. Learn more about HubSpot’s compensation philosophy. The cash compensation above includes base salary, on-target commission for employees in eligible roles, and annual bonus targets under HubSpot’s bonus plan for eligible roles. In addition to cash compensation, some roles are eligible to participate in HubSpot’s equity plan to receive restricted stock units (RSUs). Some roles may also be eligible for overtime pay. Individual compensation packages are based on a few different factors unique to each candidate, including their skills, experience, qualifications and other job-related reasons. We know that benefits are also an important piece of your total compensation package. To learn more about what’s included in total compensation, check out some of the benefits and perks HubSpot offers to help employees grow better. At HubSpot, fair compensation practices isn’t just about checking off the box for legal compliance. It’s about living out our value of transparency with our employees, candidates, and community.
We are seeking a Senior Data Engineer to join our team. In this role, you will work with various external data sources, design and maintain data pipelines, and ensure the efficiency and security of our data processes. Your contributions will help HubSpot leverage data-driven insights for effective decision-making.
This role resides within our Employee Technology group which comprises Business Systems Analysts, System Engineers, Analytic Engineers and administrators responsible for HubSpot's employee-facing application infrastructure.
In this role you will:
Data Acquisition and Ingestion
Acquire data from multiple sources using pre-built connectors and custom solutions.
Develop data ingestion pipelines to effectively manage internal and external API interactions, addressing challenges such as rate limiting, pagination, and retry mechanisms.
Process and securely ingest large datasets, ensuring encrypted data transfer and runtime credential safety.
Implement Snowpipe for data ingestion into Snowflake and utilize Snowflake’s external tables for accessing S3 data.
AWS Infrastructure and File Management
Manage and optimize data workflows within the AWS ecosystem, with a focus on S3 operations.
Manipulate S3 files upon arrival, archive older files to Glacier, and do light cleansing/alteration of files, as needed, for various workflows.
Automate AWS processes using tools like AWS Lambda, Step Functions, or custom scripts to improve efficiency and reliability.
Data Migrations and Utility Development
Develop scripts for data migrations between Snowflake instances and automate tasks using Looker’s APIs.
Create utility scripts for business needs, such as disabling inactive users, purging broken/unused Looker content, and simplifying otherwise burdensome/manual processes.
Data Modeling and Development
Build and maintain data models using dbt, following best SQL practices and data warehousing principles to ensure schema design and data integrity.
Develop data pipelines in a DAG structure, ensuring clear dependencies and performing data validation and testing.
Implement CI/CD workflows with GitHub actions/hooks for automated testing and deployment.
We are looking for people who have:
Proven experience in developing custom data ingestion pipelines and integrating with APIs, effectively managing complexities such as rate limiting and pagination.
Proficient in automating workflows using AWS Lambda, Step Functions, or comparable tools.
Extensive knowledge of databases (especially Snowflake.) Experience with Snowflake technologies such as Snowpipe, Snowpark, Data Sharing, Cloning etc., and Snowpipe usage.
Strong background in SQL for data warehouse transformations. Experience with dbt is a plus.
Solid understanding of data warehousing principles, DAG structures, and comprehensive data validation/testing frameworks.
Skilled in Python for creating utility scripts and automated workflows.
Experience with data manipulation libraries such as Pandas, PySpark, or similar technologies.
Familiarity with streaming data solutions, including Kafka, AWS Kinesis, RabbitMQ, etc
Cash compensation range: 157600-236400 USD Annually This resource will help guide how we recommend thinking about the range you see. Learn more about HubSpot’s compensation philosophy. The cash compensation above includes base salary, on-target commission for employees in eligible roles, and annual bonus targets under HubSpot’s bonus plan for eligible roles. In addition to cash compensation, some roles are eligible to participate in HubSpot’s equity plan to receive restricted stock units (RSUs). Some roles may also be eligible for overtime pay. Individual compensation packages are based on a few different factors unique to each candidate, including their skills, experience, qualifications and other job-related reasons. We know that benefits are also an important piece of your total compensation package. To learn more about what’s included in total compensation, check out some of the benefits and perks HubSpot offers to help employees grow better. At HubSpot, fair compensation practices isn’t just about checking off the box for legal compliance. It’s about living out our value of transparency with our employees, candidates, and community.
Senior Data Engineer
To see similar active jobs please follow this link: Remote Development jobs
POS-4028
We are seeking a Senior Data Engineer to join our team. In this role, you will work with various external data sources, design and maintain data pipelines, and ensure the efficiency and security of our data processes. Your contributions will help HubSpot leverage data-driven insights for effective decision-making.
This role resides within our Employee Technology group which comprises Business Systems Analysts, System Engineers, Analytic Engineers and administrators responsible for HubSpot's employee-facing application infrastructure.
In this role you will:
Data Acquisition and Ingestion
Acquire data from multiple sources using pre-built connectors and custom solutions.
Develop data ingestion pipelines to effectively manage internal and external API interactions, addressing challenges such as rate limiting, pagination, and retry mechanisms.
Process and securely ingest large datasets, ensuring encrypted data transfer and runtime credential safety.
Implement Snowpipe for data ingestion into Snowflake and utilize Snowflake’s external tables for accessing S3 data.
AWS Infrastructure and File Management
Manage and optimize data workflows within the AWS ecosystem, with a focus on S3 operations.
Manipulate S3 files upon arrival, archive older files to Glacier, and do light cleansing/alteration of files, as needed, for various workflows.
Automate AWS processes using tools like AWS Lambda, Step Functions, or custom scripts to improve efficiency and reliability.
Data Migrations and Utility Development
Develop scripts for data migrations between Snowflake instances and automate tasks using Looker’s APIs.
Create utility scripts for business needs, such as disabling inactive users, purging broken/unused Looker content, and simplifying otherwise burdensome/manual processes.
Data Modeling and Development
Build and maintain data models using dbt, following best SQL practices and data warehousing principles to ensure schema design and data integrity.
Develop data pipelines in a DAG structure, ensuring clear dependencies and performing data validation and testing.
Implement CI/CD workflows with GitHub actions/hooks for automated testing and deployment.
We are looking for people who have:
Proven experience in developing custom data ingestion pipelines and integrating with APIs, effectively managing complexities such as rate limiting and pagination.
Proficient in automating workflows using AWS Lambda, Step Functions, or comparable tools.
Extensive knowledge of databases (especially Snowflake.) Experience with Snowflake technologies such as Snowpipe, Snowpark, Data Sharing, Cloning etc., and Snowpipe usage.
Strong background in SQL for data warehouse transformations. Experience with dbt is a plus.
Solid understanding of data warehousing principles, DAG structures, and comprehensive data validation/testing frameworks.
Skilled in Python for creating utility scripts and automated workflows.
Experience with data manipulation libraries such as Pandas, PySpark, or similar technologies.
Familiarity with streaming data solutions, including Kafka, AWS Kinesis, RabbitMQ, etc
Cash compensation range: 157600-236400 USD Annually This resource will help guide how we recommend thinking about the range you see. Learn more about HubSpot’s compensation philosophy. The cash compensation above includes base salary, on-target commission for employees in eligible roles, and annual bonus targets under HubSpot’s bonus plan for eligible roles. In addition to cash compensation, some roles are eligible to participate in HubSpot’s equity plan to receive restricted stock units (RSUs). Some roles may also be eligible for overtime pay. Individual compensation packages are based on a few different factors unique to each candidate, including their skills, experience, qualifications and other job-related reasons. We know that benefits are also an important piece of your total compensation package. To learn more about what’s included in total compensation, check out some of the benefits and perks HubSpot offers to help employees grow better. At HubSpot, fair compensation practices isn’t just about checking off the box for legal compliance. It’s about living out our value of transparency with our employees, candidates, and community.
We are seeking a Senior Data Engineer to join our team. In this role, you will work with various external data sources, design and maintain data pipelines, and ensure the efficiency and security of our data processes. Your contributions will help HubSpot leverage data-driven insights for effective decision-making.
This role resides within our Employee Technology group which comprises Business Systems Analysts, System Engineers, Analytic Engineers and administrators responsible for HubSpot's employee-facing application infrastructure.
In this role you will:
Data Acquisition and Ingestion
Acquire data from multiple sources using pre-built connectors and custom solutions.
Develop data ingestion pipelines to effectively manage internal and external API interactions, addressing challenges such as rate limiting, pagination, and retry mechanisms.
Process and securely ingest large datasets, ensuring encrypted data transfer and runtime credential safety.
Implement Snowpipe for data ingestion into Snowflake and utilize Snowflake’s external tables for accessing S3 data.
AWS Infrastructure and File Management
Manage and optimize data workflows within the AWS ecosystem, with a focus on S3 operations.
Manipulate S3 files upon arrival, archive older files to Glacier, and do light cleansing/alteration of files, as needed, for various workflows.
Automate AWS processes using tools like AWS Lambda, Step Functions, or custom scripts to improve efficiency and reliability.
Data Migrations and Utility Development
Develop scripts for data migrations between Snowflake instances and automate tasks using Looker’s APIs.
Create utility scripts for business needs, such as disabling inactive users, purging broken/unused Looker content, and simplifying otherwise burdensome/manual processes.
Data Modeling and Development
Build and maintain data models using dbt, following best SQL practices and data warehousing principles to ensure schema design and data integrity.
Develop data pipelines in a DAG structure, ensuring clear dependencies and performing data validation and testing.
Implement CI/CD workflows with GitHub actions/hooks for automated testing and deployment.
We are looking for people who have:
Proven experience in developing custom data ingestion pipelines and integrating with APIs, effectively managing complexities such as rate limiting and pagination.
Proficient in automating workflows using AWS Lambda, Step Functions, or comparable tools.
Extensive knowledge of databases (especially Snowflake.) Experience with Snowflake technologies such as Snowpipe, Snowpark, Data Sharing, Cloning etc., and Snowpipe usage.
Strong background in SQL for data warehouse transformations. Experience with dbt is a plus.
Solid understanding of data warehousing principles, DAG structures, and comprehensive data validation/testing frameworks.
Skilled in Python for creating utility scripts and automated workflows.
Experience with data manipulation libraries such as Pandas, PySpark, or similar technologies.
Familiarity with streaming data solutions, including Kafka, AWS Kinesis, RabbitMQ, etc
Cash compensation range: 157600-236400 USD Annually This resource will help guide how we recommend thinking about the range you see. Learn more about HubSpot’s compensation philosophy. The cash compensation above includes base salary, on-target commission for employees in eligible roles, and annual bonus targets under HubSpot’s bonus plan for eligible roles. In addition to cash compensation, some roles are eligible to participate in HubSpot’s equity plan to receive restricted stock units (RSUs). Some roles may also be eligible for overtime pay. Individual compensation packages are based on a few different factors unique to each candidate, including their skills, experience, qualifications and other job-related reasons. We know that benefits are also an important piece of your total compensation package. To learn more about what’s included in total compensation, check out some of the benefits and perks HubSpot offers to help employees grow better. At HubSpot, fair compensation practices isn’t just about checking off the box for legal compliance. It’s about living out our value of transparency with our employees, candidates, and community.