Fedora CI and rpminspect

David Cantrell

Tim Flink

Topics

Fedora CI

rpminspect

How you can help

Fedora CI

  • Please go to the Future of automated build checks talk on Friday
  • A framework for automating tests and providing results
  • Primary goal is to know that component builds are tested, working, and ready for a compose
  • https://docs.fedoraproject.org/en-US/ci/

rpminspect

build deviation analysis tool

But first, some history...

History

Red Hat developed an internal QE tool called rpmdiff

(not rpmdiff from rpmlint)

Before virt, before containers, before the cloud

Tool is a service and performs tests on built packages

What kind of tests?

Package policy compliance

Legal checks

Kernel ABI verification

Security policies

Comparisons from one build to the next

About forty (40) tests overall

How?

Developer creates an erratum

Errata Tool schedules an rpmdiff job

rpmdiff hub (Python) verifies the builds exist

If builds exist, job is handed to a worker

Worker (Python) performs the rpmdiff

rpmdiff (Python) copies builds from NFS share

Perl script invoked to unpack the RPMs

Python script invoked to run checker

checker (in C) iterates over tests with each one spawning Perl and Python scripts

Results collected in XML format and handed back to hub

Some Problems

  • Runs on physical hardware, unable to scale
  • Legacy code base with a lot of technical debt
  • Too many moving parts
  • Cannot be run at the command line
  • Only outputs XML

Call Graph

(rpmdiff-checker run on zsh builds)

Back to rpminspect

Goals

  • Ensure package reliability
  • Ensure developer is aware of all build changes
  • Allow developers to modify packages and stay in compliance
  • Provide data to make gating decisions

Requirements

  1. Must be able to run locally
  2. Must be able to run in a container
  3. Must talk directly to build system
  4. Must be able to download builds over HTTP
  5. Must report results in multiple formats
  6. Must be usable as a command line tool
  7. Must support RPM builds and modules

Design

The service side becomes part of Fedora CI

The service listens for new builds and launches rpminspect runs

The tests are performed by rpminspect

Results collected by the service and sent to ResultsDB

Modes of Operation

  • Policy Checks
  • Build differences or deviations
    • new files appeared
    • existing files removed
    • files grew or shrunk considerably
    • new packages appeared
    • existing packages removed

Policy Checks

  • License tag
  • Header metadata
  • Political concerns
  • Forbidden language
  • Use of macros or compiler flags
  • Java bytecode major version check
  • ELF object checks
  • Security concerns (setuid, setgid, ownership)

Types of Builds

Build types consists of a grouping of RPMs and an associated SRPM for all intended architectures

  • Regular Koji RPM build
  • Module build

Support Files

  • /etc/rpminspect.conf
  • Product specific data is in the rpminspect-data-fedora package
  • rpminspect-data-generic is a template package

How to Run

  • dnf install rpminspect-data-fedora
  • man rpminspect
  • rpminspect --help
  • rpminspect [options] [previous build] [new build]
  • rpminspect -v -k -T license zsh-5.7.1-3.fc31 zsh-5.7.1-4.fc31

What Can It Do Today?

  • Speaks to Koji and downloads builds
  • Can inspect module builds
  • If only one build specified, skips deviations
  • Allows user to skip tests or only perform certain tests
  • Outputs results to stdout or a file
  • Outputs results in JSON or plain text
  • Fetch-only mode to use rpminspect to download Koji builds

Tests It Currently Performs

License tag check ¤ Check for any empty RPM payloads ¤ RPM metadata check (Vendor, badwords) ¤ Validate man pages ¤ Validate XML files ¤ ELF check (execstack, relocations) ¤ Validate *.desktop files ¤ Make sure the Release tag contains %{?dist} ¤ Validate spec file name ¤ Make sure modules contain %modularity ¤ Java bytecode major JVM version check

Call Graph

(rpminspect run on zsh builds)

Call Graphs

Contributions

  • Unit test suite
  • Integration test suite
  • Documenting the details of individual inspections
  • Designing new tests based on the Fedora packaging policy
  • Maintaining rpminspect-data-fedora

Code

  • https://github.com/dcantrell/rpminspect
  • https://github.com/dcantrell/rpminspect-data-fedora
  • Packages are in rawhide, automated builds in Copr

Questions

  • #fedora-ci on FreeNode
  • dcantrell on FreeNode
  • File github Issues
  • Email me: dcantrell@redhat.com