GistTree.Com
Entertainment at it's peak. The news is by your side.

DavHau/mach-nix: Tool to create highly reproducible Python environments

0

mach-nix – Fabricate highly reproducible python environments

Mach-nix makes it easy to carry out and share reproducible python environments or programs. Recent tools for python package deal management continually both enact no longer originate reproducibility, are no longer easy to make spend of, or require additional virtualization layers to construct them reputable. Mach-nix objectives to resolve these concerns by providing a straightforward methodology to make spend of nix, a progressive construct machine which is identified to originate recommended reproducibility and portability besides many different advantages.

Who’s this meant for?

  • Anybody who has no notion about nix however wants to retract python environments for their tasks which can maybe well be reputable and simple to reproduce.
  • Anybody who is already working with nix however wants to gash the trouble wished to carry out nix expressions for their python tasks.

Other benefits of mach-nix

  • Hardware optimizations, worship for instance SSE/AVX/FMA for tensorflow, are on hand without the must manually mess with their construct machine. (search for nixpkgs supplier)
  • Inappropriate platform enhance (tested most efficient aarch64)
  • Non-public programs or programs from a form of sources can without wretchedness be incorporated.
  • Dangle time parameters and dependencies of advanced python programs would possibly maybe maybe well furthermore be tweaked without a need to setup any construct atmosphere. It requires some recordsdata about nix, though. For examples, search for override machine.

Donate

Maintain to enhance mach-nix? A beer continuously helps ūüėČ


or

Usage from cmdline

Installation

That you just would possibly maybe both set up mach-nix by strategy of pip or by utilizing nix whenever you hold already obtained the nix package deal manager put in.

Placing in by strategy of pip

pip set up git+git://github.com/DavHau/mach-nix@3.0.0

Placing in by strategy of nix

nix-env -if https://github.com/DavHau/mach-nix/tarball/3.0.0 -A mach-nix

Dangle a virtualenv-fashion python atmosphere from a requirements.txt

mach-nix env ./env -r requirements.txt

This can generate the python atmosphere into ./env. To set off it, carry out:

The ./env directory features a portable and reproducible definition of your python atmosphere. To reuse this atmosphere on one other machine, actual copy the ./env directory
and spend nix-shell to set off it.


Generate a nix expression from a requirements.txt

mach-nix gen -r requirements.txt

…to print out the nix expression which defines a python derivation (optionally spend -o to elaborate an output file)


Usage in Nix Expression

Overall

That you just would possibly maybe call mach-nix directly from a nix expression

let
  mach-nix = import (builtins.fetchGit {
    url = "https://github.com/DavHau/mach-nix/";
    ref = "refs/tags/3.0.0";
  }) {};
in
mach-nix.mkPython {
  requirements = ''
    pillow
    numpy
    requests
  '';
}

secure extra examples beneath ./examples.md

Evolved

Mach-nix would possibly maybe maybe well furthermore be estimable tuned with additional arguments. Examples would possibly maybe maybe well furthermore be indicate in ./examples.md.

Capabilities for constructing python environments:

  1. mkPython – builds a python atmosphere for a given requirements.txt.
  2. mkPythonShell – builds a python atmosphere fair for nix-shell.
  3. mkDockerImage – builds a layered docker image containing a python atmosphere.
  4. mkNixpkgs – returns nixpkgs which is conform to the given requirements.
  5. mkOverlay – returns an overlay arrangement to construct nixpkgs conform to the given requirements .
  6. mkPythonOverrides – produces pythonOverrides to construct python conform to the given requirements.

Capabilities for constructing python programs or purposes:

  1. buildPythonPackage – construct a single python package deal from a provide code whereas automatically detecting requirements.
  2. buildPythonApplication – same as buildPythonPackage, however package deal would possibly maybe maybe well no longer be importable by a form of python programs.

buildPythonPackage and buildPythonApplication accept the an analogous arguments worship their equally named partners in nixpkgs, plus the arguments of mkPython. If name/model/requirements arguments are neglected, mach-nix makes an attempt to detect them automatically. Discover about ./examples.md.

mkPython and all a form of mk… capabilities retract precisely the following arguments:

Required Arguments:

  • requirements (string): Text disclose material of a conventional requirements.txt.

Optional Arguments:

  • suppliers (location): elaborate supplier preferences (search for examples beneath)
  • packagesExtra (list) Add additional programs. Can have tarball-URLs or paths of python provide code, programs built by strategy of mach-nix.buildPythonPackage, or R Capabilities.
  • _ (location): spend underscore argument to without wretchedness adjust arbitrary attributes of programs. As an instance so that you just could add built inputs spend _.{package deal}.builtInputs.add = [...]. Or to determine patches spend _.{package deal}.patches = [...].
  • overridesPre (list): (developed) list of pythonOverrides to apply sooner than the machnix overrides. Employ this to consist of additional programs which will then be selected contained in the requirements
  • overridesPost (list): (developed) list of pythonOverrides to apply after the machnix overrides. Employ this to fixup programs.
  • tests (bool): Whether or no longer to permit tests (default: counterfeit)
  • _providerDefaults (location): builtin supplier defaults. Disable them by passing {}

Configure Suppliers

Suppliers allow you configure the starting keep to your programs on a granular basis.

The next 3 suppliers are on hand since model 2.0.0:

  1. nixpkgs: Presents programs directly from nixpkgs without modifying their sources. Has most efficient about a variations on hand, however has a excessive success price and all the nix supported functions, worship SSE/AVX/FMA for tensorflow for instance.
  2. sdist: Presents all package deal variations on hand from pypi which enhance setuptools and builds them by strategy of nixpkgs overlays wherever imaginable to resolve exterior dependencies. It still supports the nixpkgs explicit functions no topic which package deal model is chosen. But likelihood is increased for a construct to fail than with the nixpkgs supplier.
  3. wheel: Presents all linux worship minded wheel releases from pypi. Wheels can have binaries. Mach-nix autopatches them to work on nix. Wheels are paunchy posthaste to set up and work reasonably reputable. Therefore this supplier is most neatly-most neatly-liked by default.

Mach-nix builds environments by mixing programs from all 3 suppliers. You to deem which suppliers ought to be most neatly-most neatly-liked for which programs, or which suppliers need to not be normal in any appreciate.
The default most neatly-most neatly-liked portray of suppliers is wheel, sdist, nixpkgs.

Suppliers would possibly maybe maybe well furthermore be disabled/enabled/most neatly-most neatly-liked worship in the following examples:

  • A supplier specifier worship "wheel,sdist,nixpkgs" methodology, that the resolver will first attempt to meet the requirements with candidates from the wheel supplier. If a resolution is highly no longer going or a package deal doesn’t provide a wheel launch, it falls encourage to sdist/nixpkgs for a minimal different of programs. In weird and wonderful this could maybe well lift as many programs from wheel as imaginable, then sdist, then nixpkgs.

  • "nixpkgs,sdist" methodology, that nixpkgs candidates are most neatly-most neatly-liked, however mach-nix falls encourage to construct from provide (sdist). wheel just isn’t any longer listed and therefore wheels are disabled.

A full supplier config handed to mach-nix appears to be like to be like worship this:

{
 # The default for all programs which can maybe well be no longer specified explicitly
 _default = "nixpkgs,wheel,sdist";

 # Mumble settings per package deal
 numpy = "wheel,sdist";
 tensorflow = "wheel";
}

Mach-nix will continuously fulfill the requirements.txt fully with the configured suppliers or fail with a ResolutionImpossible error.

If a mach-nix construct fails, Most of the instances it could maybe well furthermore be resolved by actual switching the supplier of a package deal, which is easy and doesn’t require writing quite quite a bit of nix code. For some extra advanced scenarios, checkout the ./examples.md.

Why nix?

On the total of us rely on multiple layers of a form of package deal management tools for constructing their tool environments. These tools are continually no longer neatly integrated with every a form of and fabricate no longer provide solid reproducibility. Example: It’s essential maybe well be on debian/ubuntu and spend APT (layer 1) to set up python. Then you definately spend venv (layer 2) to overcome some of your layer 1 boundaries (no longer being able to hold multiple variations of the an analogous package deal put in) and afterwards you are utilizing pip (layer 3) to set up python programs. You note that even after pinning your entire requirements, your atmosphere behaves in a different plan on your server or your colleagues machine attributable to their underlying machine differs from yours. You launch up utilizing docker (layer 4) to overcome this self-discipline which adds additional complexity to the total course of and affords you some wrong boundaries all the plan in which by strategy of pattern. It is advisable to configure your IDE’s docker integration and plenty others. Regardless of all the trouble you put in, still the subject just isn’t any longer fully solved and now and again your construct pipeline actual breaks and you’ve got to repair it manually.

In distinction to that, the nix package deal manager affords a from floor up a form of methodology to construct tool programs. Because of it is purly purposeful methodology, nix doesn’t require additional layers to construct your tool reputable. Tool environments built with nix are identified to be reproducible, and transportable, which makes many processes all the plan in which by strategy of pattern and deployment more straightforward. Mach-nix leverages that doable by abstracting away the complexity focused on constructing python environments with nix. Usually it actual generates nix expressions for you.

How does mach-nix work?

The weird and wonderful mechanism would possibly maybe maybe well furthermore be damaged down into Dependency resolution and Generating a nix expression:

Dependency resolution

Mach-nix features a dependency graph of virtually all python programs on hand on pypi.org. This allows mach-nix to resolve dependencies offline interior seconds.

The dependency graph recordsdata would possibly maybe maybe well furthermore be realized here: https://github.com/DavHau/pypi-deps-db


The dependency graph is updated on a day after day basis by this location of tools: https://github.com/DavHau/pypi-crawlers

Regardless of this graph being updated continuously, mach-nix continuously pins one explicit model of the graph to substantiate that reproducibility.

As core for the resolving resolvelib is normal: https://github.com/sarugaku/resolvelib

Mach-nix supports multiple suppliers to retrieve python programs from. The user can specify which suppliers ought to be most neatly-most neatly-liked. Capabilities from a form of suppliers would possibly maybe maybe well furthermore be blended.

Generating a nix expression

After all python dependencies and their suppliers had been certain by the dependency resolver, mach-nix will generate a nix expression defining your python atmosphere.

Particular person python programs are both built by overriding an present package deal definition from nixpkgs, or by creating the package deal from scratch by strategy of nixpkgs’ buildPythonPackage. Which formulation is normal relies on the supplier of a package deal and if it is already packaged in nixpkgs.

The spend of nixpkgs as a unfriendly has the following benefits:

  1. Non-python Dependencies:


    Many python programs hold non-python dependencies worship a form of C libraries. Mach-nix can resolve those dependencies by taking the construct inputs from python package deal definitions in nixpkgs.
  2. Particular functions:
    Some python programs would possibly maybe maybe well furthermore be built with particular functions, worship for instance SSE/AVX/FMA enhance in tensorflow. The nixpkgs variations of those python programs continually consist of these functions.
  3. Nix explicit fixes:


    Some python programs would possibly maybe maybe well want some additional modification to work with nix. Those are already executed in nixpkgs.

If a package deal is built by overriding nixpkgs, the following attributes are modified:

  • src: updated to the desired model
  • name: modified to verify the fresh model
  • buildInputs: modified with mach-nix certain python deps
  • propagatedBuildInputs: non-python deps of former definition + mach-nix certain python deps
  • doCheck: location to counterfeit by default if no longer specified by user
  • doInstallCheck: location to counterfeit by default if no longer specified by user

Contributing

Contributions to this mission are welcome in the fabricate of GitHub PRs. Whilst you are planning to construct any in actual fact broad changes, you ought to still first indicate your plans in a GitHub self-discipline so it could maybe well furthermore be talked about.

Boundaries

  • Within the intervening time mach-nix does no longer provide any functionality which supports you in publishing python tasks, worship Poetry does for instance.

Different / Similar Tool:

Read More

Leave A Reply

Your email address will not be published.