release-engineering/repo-autoindex
1
0
Fork 0
mirror of https://github.com/release-engineering/repo-autoindex.git synced 2025-02-23 21:52:52 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 1
repo-autoindex/README.md
Rohan McGovern eac74ec1e4 Further reduce memory usage on large yum repos [RHELDST-20453] 
The Fetcher type was designed to return a 'str'.
That wasn't a good idea because it implies that every fetched file must
be loaded into memory completely. On certain large yum repos,
decompressed primary XML can be hundreds of MB, and it's not appropriate
to require loading that all into memory at once.

Make it support a file-like object (stream of bytes). Since the SAX
XML parser supports reading from a stream, this makes it possible to
avoid loading everything into memory at once.

A test of repo-autoindex CLI against
/content/dist/rhel/server/7/7Server/x86_64/os showed major
improvement:

- before: ~1200MiB
- after:    ~80MiB

Note that achieving the full improvement requires any downstream users
of the library (e.g. exodus-gw) to update their Fetcher implementation
as well, to stop returning a 'str'.
2023-09-21 11:05:21 +10:00

1.9 KiB
Raw Blame History

repo-autoindex

Generate static HTML indexes of various repository types

Build Status Coverage Docs PyPI

Overview

repo-autoindex provides a minimal CLI and Python library to generate static HTML indexes for certain types of content, such as yum repositories.

pip install repo-autoindex
REPO_URL=$(curl -s 'https://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f36&arch=x86_64' | egrep '^http' | head -n1)
repo-autoindex $REPO_URL
xdg-open index.html

See the manual for more information about the usage of repo-autoindex.

Changelog

v1.2.0 - 2023-09-22

  • Support streamed fetching to reduce memory usage when fetching large files.

v1.1.2 - 2023-09-18

  • Add py.typed to make package PEP 561 compliant / enable downstream type-checking.

v1.1.1 - 2023-04-12

  • Fix handling of kickstart repositories with no checksums in treeinfo.

v1.1.0 - 2023-04-04

  • Added limited support for kickstart repositories.

v1.0.2 - 2022-10-21

  • Reduced memory usage when handling large yum repositories.

v1.0.1 - 2022-08-15

  • Use correct SPDX license identifier in package metadata.

v1.0.0 - 2022-08-15

  • Initial stable release.

License

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.