Python Out Of Core Numpy Developer Full-time

(also available as a 3-6 month traineeship)

In 2014 Nexedi developed a technology called Wendelin.core which provides out-of-core python ndarrays that can be shared transparently across different nodes of a cluster of python runtimes. With Wendelin.core, python can be used natively for big data without relying on other languages or runtimes. Wendelin.core is already being used in production for example for monitoring offshore wind turbines and detecting anomalies. As most use cases of Wendelin.core involve third-party libraries such as NumPy or scikit-learn who run methods “not aware” of available memories, a key challenge for us is to ensure the libraries we deploy in production perform under heavy data loads.

Nexedi is looking for a candidate interested in improving libraries utilized by Wendelin and wendelin.core (mostly NumPy, scikit-learn to some degree, other depending on specific implementation) to reduce the number of memory allocations or copies made internally. This task may require to modify default algorithms that use array allocations or replace them with algorithms that modify data in-place. It may also require to allocate explicitly out-of-core ndarrays whenever there is no better way and contribute any changes made back to upstream improving libraries utilized for the community.

Opportunities

• Contribute to NumPy, scikit-learn and other libraries used in Wendelin implementations.
• Improve Wendelins and wendelin.core’s capabilities to analyse TB of data.
• Help harden our stack of free software solutions.

Responsibilities

• Learn to master and improve Wendelin, Jupyter extensions and libraries.
• Contribute memory-aware data handlers to popular Python libraries.
• Add your knowhow and experience to industrial implementations.

Requirements

• Passionate, self-driven.
• Willingness to contribute to an open source ecosystem and the Free Software community.
• Very good programming skills in Python.
• Very good skills in algorithms and experience with NumPy, scikit-learn.
• Good software development skills (version control, testing, debugging).
• Good command of English.