Senior Site Reliability Engineer (Distributed Storage) Full-time

Summary

We are looking for a Senior Site Reliability Engineer to support and develop the platform serving the world’s favorite encyclopedia to millions of people around the globe. Wikipedia and its sister projects are powered strictly by Free and Open Source software with MediaWiki at its core surrounded by an ecosystem of microservices in PHP, NodeJS, Python, Go and Java.

The Wikimedia community generates a sizable amount of data, including freely-licensed media. As part of our team you would have the opportunity to solve storage, scaling, access and backup challenges Wikipedias’ scale provides.

We currently use Openstack Swift as our main object-storage system and Cassandra as the dominant NoSQL solution. We are interested in finding new and improved ways of utilizing the capabilities of the existing solutions as well as researching and potentially implementing other similar products.

We are a globally distributed and diverse team of engineers with a drive to explore, experiment and embrace new technologies. If you find what we do interesting, if you are up to the challenge of improving the reliability and delivery of one of the Internet’s top 10 websites, and you enjoy the idea of working in a remote-first role, we may just be the right place for you.

Responsibilities

Design, improve and operate the distributed storage and NoSQL systems serving Wikimedia projects
Research, test and benchmark new storage and NoSQL solutions
Respond to common database issues, following documented procedures
Collaborate with SRE and other teams across the organization to ensure their specific storage, backup and DR needs are met on an ongoing basis
Perform day-to-day operational/DevOps tasks on Wikimedia’s public facing infrastructure (deployment, maintenance, configuration, troubleshooting)
Implement and utilize configuration management and deployment tools
Assist in the architectural design of new services and making them operate at scale
Monitor systems, services and service clusters, optimize performance and resource utilization
Assist in or lead incident response, diagnosis and follow-up on system outages or alerts across Wikimedia’s production infrastructure
Mentor peers in your areas of technical and operational strength
Share our values and work in accordance with them
Qualifications
3+ years experience in an SRE/Operations/DevOps role as part of a team
Experience with (Free and Open Source) distributed storage products (Openstack Swift, Ceph, etc.)
Experience with NoSQL products (Cassandra, Redis, Elasticsearch, etc.)
Comfortable with shell and scripting languages used in an SRE/Operations engineering context (Python, Go, Bash, Ruby, etc.)
History of automating tasks and processes, identifying process gaps and finding automation opportunities
Strong English language skills and ability to work independently, as an effective part of a globally distributed team
B.S. or M.S. in Computer Science or equivalent work experience
Pluses
Familiarity with database administration and support, debugging query performance and schema design
Experience managing data backups and knowledge of popular Open Source backup management software (bacula, amanda, rdiff-backup/duplicity, etc…)
Experience with centralized storage systems (SAN and NAS) and/or tape storage systems
Low level systems troubleshooting and debugging (CPU/memory profiling, C/C++ experience, in-depth Linux knowledge)
Experience with LAMP stack technologies (PHP/HHVM, memcached/redis, MySQL) – MediaWiki familiarity would be great!
Developing Free and Open Source software, or being part of an open-source community
The Wikimedia Foundation is…

…the nonprofit organization that hosts and operates Wikipedia and the other Wikimedia free knowledge projects. Our vision is a world in which every single human can freely share in the sum of all knowledge. We believe that everyone has the potential to contribute something to our shared knowledge, and that everyone should be able to access that knowledge, free of interference. We host the Wikimedia projects, build software experiences for reading, contributing, and sharing Wikimedia content, support the volunteer communities and partners who make Wikimedia possible, and advocate for policies that enable Wikimedia and free knowledge to thrive. The Wikimedia Foundation is a charitable, not-for-profit organization that relies on donations. We receive financial support from millions of individuals around the world, with an average donation of about $15. We also receive donations through institutional grants and gifts. The Wikimedia Foundation is a United States 501©(3) tax-exempt organization with offices in San Francisco, California, USA.

As an equal opportunity employer, the Wikimedia Foundation values having a diverse workforce and continuously strives to maintain an inclusive and equitable workplace. We encourage people with a diverse range of backgrounds to apply. We do not discriminate against any person based upon their race, traits historically associated with race, religion, color, national origin, sex, pregnancy or related medical conditions, parental status, sexual orientation, gender identity, gender expression, age, status as a protected veteran, status as an individual with a disability, genetic information, or any other legally protected characteristics.

If you are a qualified applicant requiring assistance or an accommodation to complete any step of the application process due to a disability, you may contact us at recruiting@wikimedia.org or (415) 839-6885.

U.S. Benefits & Perks*

Fully paid medical, dental and vision coverage for employees and their eligible families (yes, fully paid premiums!)
The Wellness Program provides reimbursement for mind, body and soul activities such as fitness memberships, baby sitting, continuing education and much more
The 401(k) retirement plan offers matched contributions at 4% of annual salary
Flexible and generous time off – vacation, sick and volunteer days, plus 22 paid holidays – including the last week of the year.
Family friendly! 100% paid new parent leave for seven weeks plus an additional five weeks for pregnancy, flexible options to phase back in after leave, fully equipped lactation room.
For those emergency moments – long and short term disability, life insurance (2x salary) and an employee assistance program
Pre-tax savings plans for health care, child care, elder care, public transportation and parking expenses
Telecommuting and flexible work schedules available
Appropriate fuel for thinking and coding (aka, a pantry full of treats) and monthly massages to help staff relax
Paid travel to Wikimedia Foundation events all around the world!
Equipment including a laptop, monitor, plus a one-time stipend to cover any additional needs to make sure you have the best work experience
Great colleagues – diverse staff and contractors speaking dozens of languages from around the world, fantastic intellectual discourse, mission-driven and intensely passionate people *Please note that for remote roles located outside of the U.S., we defer to our PEO to ensure alignment with local labor laws.