1 person found this helpful
We changed how the index pages are generated, but that shouldn't impact mrepo/reposync. Can you provide your mrepo config for one of the public-yum.oracle.com repos you're mirroring? Also, what time do you trigger your mirror, out of interest? My local mirrors that use the OTN mirroring script, i.e. reposync, are all still fine, so we need to work out what mrepo did.
Also, if you run mrepo again, does it start to download the packages?
P.S. it's Oracle OpenWorld this week, so replies could be delayed!
Thanks for the reply. The sync runs at ~0500 AEST.
I was using the 'http' method, but have now upgraded mrepo to 0.8.8 and moved to an OL6 box so I can have reposync. Kicked it off last night and everything seems to be working again.
Running mrepo again before the above changes didn't make any difference.
FWIW and any one in the future relevant sections of mrepo configuration that I added/changed:
reposync-cleanup = no
reposync-newest-only = yes
reposync-options = -n
name = Oracle Linux 6 (x86_64)
release = 6
metadata = repomd
latest = reposync://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64
addons = reposync://public-yum.oracle.com/repo/OracleLinux/OL6/addons/x86_64
### Note the 'reposync' use to be 'http' in these last 2 lines.
Happy mirroring, and hopefully someone else finds this post useful.
1 person found this helpful
Ok, that makes sense now -- we had to change the way the directory indexes were created because we're about to turn on a CDN for public-yum.oracle.com. This means the HTTP download option in mrepo probably will no longer work. We obviously still support the reposync option (which is, btw, the only method we officially support).
Though... there is more to this.
The packages are getting sync'd into <mrepo dir>/getPackage/xyz.rpm, instead of <mrepo dir>/xyz.rpm, doing a simple mv * ../ works and the mrepo generation step works again, but this feels a little unnecessary...
Is there one other trick I am missing?
mrepo should still work even though the packages are getting saved into getPackage/ -- that's actually where the packages are stored on public-yum.oracle.com (and behind the scenes on ULN proper too). I've tested the latest mrepo doing a sync directly with reposync and it worked fine for me even with the getPackage stuff.
I'm currently at OpenWorld, so I don't have a lot of free time to test this, but I'll give mrepo another spin next week to make sure it's all working 100% with our CDN as well.
You are right! Spoke to soon. All is good in the world again
@AviMiller thanks again for your help, always awesome!
You're welcome. I did a bunch of work with mrepo (including submitting a pull request to the GitHub repo to enable ULN support) but it seems like mrepo development is stalled. You may want to check out our ULN mirroring script on OTN, though that requires a ULN account. Alternatively, both Spacewalk and Pulp work great against public-yum.oracle.com and both of those will download the yum security metadata as well, something that mrepo doesn't do.
I just want to note that syncing the latest OL6 repo with Pulp is currently broken. I have opened a ticket with the Pulp team, but this may be something that you want to look at as well:
It would appear that Oracle modifies the errata and does not include a reference to the SRPM. Upstream (RHEL) errata data does contain 'src.'
Note that we don't actually modify any errata: we build it ourselves. When you say we're missing a reference to the SRPM, can you be more specific about where it should be in the XML? The SRPM parameter is supposed to be optional in updateinfo.xml, so this sounds more like a bug in Pulp than a problem with the way we generate the metadata. However, if we can improve our metadata, I'm all ears!
See this as an example:
Each package as an "src" attribute, ie:
<package arch="noarch" name="walrus" release="1" src="http://www.fedoraproject.org" version="0.71">
The OL updateinfo.xml is missing this attribute.
If this is only happening with Pulp, why is it something that needs to be fixed in Oracle Linux or updateinfo.xml? It rather looks like something needs to be fixed in Pulp. Maybe its not parsing the information correctly.
Which is why I opened a bug with Pulp. I am simply letting Oracle know.. maybe their lack of the "src" attribute in updateinfo.xml is not intentional.
Right, I thought it would be a pointer to the SRPM, but it's just a URL to the source provider. Weird attribute. We could certainly shove a src attribute into our updateinfo.xml, but I'm not sure it'll be high on our list at all. Do let us know if/when Pulp decides to fix this, though.
If I remember correctly, the updateinfo.xml.gz file was actually missing in the Oracle Public Yum repository, causing the yum security plug-in to fail. The issue was apparently fixed around May this year.
The file http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/repodata/updateinfo.xml.gz actually contains a lot of Oracle added information. So whatever is used to generate the file might indeed be missing the src package attribute.
http://en.opensuse.org/openSUSE:Standards_Rpm_Metadata_UpdateInfo seems to explain the format of the file, which lists the src attribute.
So maybe this is indeed something the Oracle Linux team might want to look into. Though I wonder why this has not been an issue so far.
What was your reason for hijacking this discussion? Your problem has obviously nothing to do with the subject of the thread or problem of the discussion owner.