Job description

Jan is a desktop application that turns computers into thinking machines, enabling open-source LLMs and remote API AIs on Windows, Mac, or Linux. Since its January 2024 release, it has been downloaded over 500,000 times. With a vision to redefine human-machine collaboration, Jan aims to revolutionise our interaction with technology. It ensures complete ownership, privacy, and full customisation for its users.

Jan is looking for a Machine Learning Software engineer can help run Jan on a wide variety of consumer-grade hardware to server-grade hardware, to support the 300,000 people who have downloaded Jan since 1st Jan 2024.

Responsibilities

Ensuring Jan’s inference and training to run stably on the following dimensions:

CPU Hardware:
- Intel consumer and Xeon (incl. AVX-2, AVX-512, AVX instructions).
- AMD Ryzen.
- Apple Silicon.
GPU/ AI specific Hardware:
- Nvidia (CUDA, Triton Inference Server using either vLLM/ TensorRT-LLM Backends).
- AMD (ROCm), AMD Ryzen AI.
- Intel Arc (sycl, Big DL, Intel Extensions for Transformers), Intel NPU.
- Apple Metal/ CoreML.
Cross-platform model format (e.g. GGUF, Pytorch).
Operating Systems (Windows, MacOS, Linux).
Popular Inference engines (llama.cpp, vLLM, TensorRT-LLM).
Popular Training framework (Pytorch, MLX).
[Bonus] Edge training/ fine-tuning, micro training/ federated training on various hardware and software AI stacks.

Requirements

Demonstrated software engineering skill and proven track in Python, C++.
Ability to be communicative in English.
Familiar with development tools and workflows such as Git, CI/CD, Agile.
Experience with hardware optimization (CPU, GPU, NPU).
Experience but not limited to LLMs, including popular foundation models like Llama2, Mistral, GPT2.
[Bonus] Experience running LLMs/ AI models in production in server/ edge.

Benefits

We pay an “all-in” pay and you will cover your own insurance/medical from the amount.
14 days leave (and unlimited sick days).
Annual equipment budget (once 2 month probation has been completed).

Required profile