Commit graph

219 commits

Author SHA1 Message Date
Rohan McGovern
eac74ec1e4 Further reduce memory usage on large yum repos [RHELDST-20453]
The Fetcher type was designed to return a 'str'.
That wasn't a good idea because it implies that every fetched file must
be loaded into memory completely. On certain large yum repos,
decompressed primary XML can be hundreds of MB, and it's not appropriate
to require loading that all into memory at once.

Make it support a file-like object (stream of bytes). Since the SAX
XML parser supports reading from a stream, this makes it possible to
avoid loading everything into memory at once.

A test of repo-autoindex CLI against
/content/dist/rhel/server/7/7Server/x86_64/os showed major
improvement:

- before: ~1200MiB
- after:    ~80MiB

Note that achieving the full improvement requires any downstream users
of the library (e.g. exodus-gw) to update their Fetcher implementation
as well, to stop returning a 'str'.
2023-09-21 11:05:21 +10:00
github-actions[bot]
02501a7b36
Merge pull request #65 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-09-19 20:22:40 +00:00
Rohan McGovern
c557a61876
Merge pull request #66 from rohanpm/add-py.typed
Add py.typed for PEP 561
2023-09-19 07:31:33 +10:00
Rohan McGovern
efb595d624 Add py.typed for PEP 561
This library includes inline type hints, but per PEP 561 this must be
indicated by including a "py.typed" marker file, otherwise tools like
mypy will not make use of the type hints when checking downstream
projects.
2023-09-18 13:28:02 +10:00
GitHub Actions
e3ba70a530 chore: scheduled poetry update 2023-09-17 20:21:36 +00:00
github-actions[bot]
2456127e36
Merge pull request #64 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-09-12 20:22:34 +00:00
GitHub Actions
f88a906f2b chore: scheduled poetry update 2023-09-10 20:21:30 +00:00
github-actions[bot]
090d4b1442
Merge pull request #63 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-09-05 20:21:46 +00:00
GitHub Actions
79f3203441 chore: scheduled poetry update 2023-09-03 20:21:30 +00:00
github-actions[bot]
a701f6accc
Merge pull request #62 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-08-22 20:22:17 +00:00
GitHub Actions
bfdadfe53f chore: scheduled poetry update 2023-08-20 20:21:05 +00:00
github-actions[bot]
6064a8e8fd
Merge pull request #61 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-08-15 20:22:09 +00:00
GitHub Actions
f4eae98e91 chore: scheduled poetry update 2023-08-13 20:21:14 +00:00
github-actions[bot]
516dc5140a
Merge pull request #60 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-08-08 20:22:36 +00:00
GitHub Actions
0c963d3cc3 chore: scheduled poetry update 2023-08-06 20:21:13 +00:00
github-actions[bot]
865451d32f
Merge pull request #59 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-08-01 20:22:23 +00:00
GitHub Actions
5bf42111ab chore: scheduled poetry update 2023-07-30 20:21:25 +00:00
github-actions[bot]
85abb7ffb5
Merge pull request #58 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-07-25 20:22:07 +00:00
GitHub Actions
d5eab6a5f8 chore: scheduled poetry update 2023-07-23 20:21:52 +00:00
github-actions[bot]
ccc63f8bce
Merge pull request #57 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-07-18 20:23:15 +00:00
GitHub Actions
a058b5ae62 chore: scheduled poetry update 2023-07-16 20:23:12 +00:00
github-actions[bot]
6cd49e7ea7
Merge pull request #56 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-07-11 20:23:55 +00:00
GitHub Actions
4c49a62961 chore: scheduled poetry update 2023-07-09 20:23:11 +00:00
github-actions[bot]
6afd5409fd
Merge pull request #55 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-07-04 20:24:28 +00:00
GitHub Actions
a7204df4cc chore: scheduled poetry update 2023-07-02 20:23:03 +00:00
github-actions[bot]
c6411babe8
Merge pull request #54 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-06-27 20:24:09 +00:00
GitHub Actions
9c37bdf24a chore: scheduled poetry update 2023-06-25 20:23:48 +00:00
github-actions[bot]
031ed5a98c
Merge pull request #53 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-06-20 20:21:51 +00:00
GitHub Actions
e817ba52e3 chore: scheduled poetry update 2023-06-18 20:22:18 +00:00
github-actions[bot]
46753f6459
Merge pull request #52 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-06-13 20:22:40 +00:00
GitHub Actions
a6b455563d chore: scheduled poetry update 2023-06-11 20:22:02 +00:00
github-actions[bot]
7391b2ecff
Merge pull request #51 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-06-06 20:22:37 +00:00
GitHub Actions
e9739ae501 chore: scheduled poetry update 2023-06-04 20:21:49 +00:00
github-actions[bot]
466a0c3880
Merge pull request #50 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-05-30 20:22:31 +00:00
GitHub Actions
44583eed20 chore: scheduled poetry update 2023-05-28 20:22:02 +00:00
github-actions[bot]
f1dcd70547
Merge pull request #48 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-05-23 20:22:33 +00:00
GitHub Actions
1b44f642f6 chore: scheduled poetry update 2023-05-21 20:22:07 +00:00
GitHub Actions
a000221d61 Merge remote-tracking branch 'origin/main' into deps/poetry-update 2023-05-21 20:21:37 +00:00
Rohan McGovern
dd98f3f432
Merge pull request #49 from rohanpm/mypy-20230515
Update and fix mypy with latest type hints again
2023-05-16 07:15:46 +10:00
Rohan McGovern
6ffbe4736c Update and fix mypy with latest type hints again
3f478e76f7 added a "type: ignore" here due to a change in
typeshed. The commit message mentioned that the type hint may have been
wrong.

It looks like that was fixed in
https://github.com/python/typeshed/pull/9919/files,
so it's necessary to also remove the "type: ignore" now.
2023-05-15 08:45:01 +10:00
Rohan McGovern
6a88656fa0
Merge pull request #44 from crungehottman/codeowners
Add CODEOWNERS file
2023-05-15 08:39:05 +10:00
GitHub Actions
e25f349540 chore: scheduled poetry update 2023-05-14 20:22:09 +00:00
github-actions[bot]
5c35886354
Merge pull request #47 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-05-09 20:22:44 +00:00
GitHub Actions
00ffe18c1d chore: scheduled poetry update 2023-05-07 20:22:13 +00:00
github-actions[bot]
3ca2b6bb91
Merge pull request #46 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-05-02 20:22:42 +00:00
GitHub Actions
b4ad9ad1e7 chore: scheduled poetry update 2023-04-30 20:22:03 +00:00
github-actions[bot]
070005e20e
Merge pull request #45 from release-engineering/deps/poetry-update
chore: scheduled poetry update
2023-04-25 20:23:11 +00:00
GitHub Actions
16e43f2a75 chore: scheduled poetry update 2023-04-23 20:22:31 +00:00
Rohan McGovern
4b37dc745a
Merge pull request #43 from rohanpm/mypy-fix
Fix type check with latest mypy/typeshed
2023-04-20 07:23:04 +10:00
Caleigh Runge-Hottman
a0c2edcc93 Add CODEOWNERS file 2023-04-19 09:24:56 -04:00