Software Engineer - Platform InfraCore

This is a remote position and we're considering candidates in the USA only.

Software Engineer - Platform InfraCore

About our Platform (at Grafana Labs):

Grafana Cloud moves millions of metrics, log lines, and traces per second from our customers' environments into a highly available, low-latency stack that processes and stores this data, and serves them to dashboards and alerting tools. We aim to grow this to hundreds of millions per second, and it's critical that as we grow, we improve our performance, increase our reliability, and, of course, do it efficiently and effectively.

The Internal Engineering Platform (IEP) delivered by the Platform department provides application engineers with the tools, systems and Kubernetes clusters they need to build, deploy and run their workloads. Platform roles at Grafana Labs have an eye for engineers with a passion for performance and reliability, and who enjoy taking projects from conception to production. We organize ourselves into squads to allow focus on: cloud infrastructure and capacity management; security; engineering productivity; monitoring and sustainability; and US Federal compliance.

Because we deploy production services, we have on-call rotations to ensure the health of the system. Everyone at Grafana Labs tries to incorporate and use our product line up into their day-to-day, so being on call is an important way to understand our system and how people use our products.

About the squad & the role:

We are hiring for the Platform InfraCore squad. This squad provides and owns automation of the provisioning of CSP resources, including networking, Kubernetes clusters and specific CSP resources required by our application teams.

We’re part of a Platform Engineering group that manages infrastructure for the teams that are building some of the most cherished tools - Grafana, Mimir, Loki, Tempo, Pyroscope to name a few.

Who Are We Looking For?

• You enjoy working with engineers, as well as with the management structures that are there to support you and enable you and your team to do your very best.

• You are comfortable working in a remote-first company; communication is key. For us, working together means being collaborative, friendly, kind, and respectful. We operate by consensus, you can contribute to a discussion but then commit to the team decision.

• As such, being such a highly distributed company, means we would love someone who is keen on working with distributed systems, too.

• You are eager to learn and grow. There is a lot of room for growth and development, and the team has quite a lot of knowledge to share for those who are wanting to learn.

• You approach development holistically. The team owns the full life cycle of our code; from writing design docs, to looking at developer feedback, and integration testing. We appreciate engineers who enjoy looking at the big picture, and also notice the details of the brush strokes. The Platform team mainly works with Go, Python, and Shell.

• You have experience with operating your code. Since a lot of operators and developers use our software, having some grounding in both of these spaces really helps us with building better platforms for our users.

Some skills/experience we like to see:

Responsibilities

• Kubernetes cluster provisioning and lifecycle management.

• Management of cluster networking components: load balancing, NAT, DNS, CNIs, network policies, private connectivity for customers, cross-cluster communication.

• Management of scheduling and autoscaling.

• Maintaining Crossplane compositions and Terraform modules for CSP resources common to our users. As well the management of versioning and compatibility for Crossplane and Terraform core as well as providers.

• Work with our users (Grafana Cloud application teams) to help understand their needs and ensure we’re investing in the right capabilities.

• Participation in the Platform department Infrastructure wing on-call rotation.

Nice to have

• You’ve worked in or on open source, or other community-based projects previously. At Grafana Labs, “OSS is in our DNA”.

• Experience with a few CSPs. We run Grafana Cloud on AWS, GCP, and Azure using each’s managed Kubernetes service - EKS, GKE, AKS.

• Experience operating and managing workloads on Kubernetes. We use Tanka for configuration management with Jsonnet.

• Familiarity with Kubernetes scheduling and projects like Karpenter.

• Terraform and/or Crossplane experience. We have mixed usage - each has its strengths.

• Enjoys programming in Go! We love building our own tools, utilities, exporters, etc. that suit our needs and otherwise don’t exist (and open sourcing them).

• Maybe there’s something missing in this job description that you think you might bring to our table? Feel free to let us know!

A few of our upcoming projects:

• Improving K8s efficiency. Considerations include processor architecture, CSP capacity, machine selection. Balancing utilization with idle compute to ensure suitable performance and cost.

  • Improving network policy adoption and interfaces. Make it easier for our users to create the perfect network policies and tests for their workloads.

  • Refining/expanding SLOs. Our users need to know exactly what to expect from their IEP. We have explicit measures to indicate when we are and are not on target. We’re continually expanding the picture.

In the US, the Base compensation range for this role is $123,933 - $148,719. Actual compensation may vary based on level, experience, and skillset as assessed in the interview process. Benefits include equity, bonus (if applicable) and other benefits listed here.

*Compensation ranges are country-specific. If you are applying for this role from a different location than listed above, your recruiter will discuss your specific market’s defined pay range & benefits at the beginning of the process.