Newznab backfilling

I’ve recently been checking out newznab plus. I’ve found that while it’s nice as it slowly builds, it’s a bit annoying not having any older posts already loaded. Fortunately nn+ (nnp, newznab plus) comes with an option for backfilling. The instructions are useful but I’d like to expand on them to make it less of a hassle to update more than one particular group.

Please note that I’ve only been using nn+ for a few days now so there may indeed be easier ways to do things but this is the best way I’ve found so far. For example, I’m not exactly sure what the update_binaries_threaded.php file does but I don’t believe it does what I’m discussing here. I’m guessing from the “threaded” that it simply makes multiple threads of the same process to make the entire task faster.

The following is taken from http://newznab.readthedocs.org/en/latest/readme/#backfilling-groups

Backfilling Groups
Since most usenet providers have 800+ days of retention indexing all that information in one shot is not practical. Newznab provides a backfill feature that allow you to index past articles once your initial index has been built. To use the feature first set the back fill days setting in the group(s) to be backfilled to the number of day you wish to go back, making sure to set it higher than the number of days listed in the first post column. Once set run the backfill.php script in misc/update_scripts. Groups can be backfilled to a particular date using the script misc/update_scripts/backfill_date.php using the syntax: php backfill_date.php 2011-05-15 alt.binaries.groupname.here You can use the _threaded version of this script if on linux. For more information on backfilling, see Update Scripts.

First off, what we need to do is to activate any groups we want. More than likely you’ve already done this. What you’ll want to do in order to backfill those groups is to change the ‘Backfill Days’ entry. This can be done through the webpage but if you are wanting to backfill all of them then this following command will make it much faster than manually clicking through a dozen or more groups. You’ll need to be on the command line and then log into mysql

mysql -u root -p
Enter password:

mysql> use newznab; (or whatever your database name is)
mysql> update groups set backfill_target=365 where active=1;

You have now just changed all active groups to have a backfill of 365 days (1 year). Now we just need to run our backfill_date.php on each group. If we’ve got dozens of groups or more though this can be tedious to repeat over and over with each group. So instead we are going to get a list of all the active groups by running the following command:

mysql -u root -p newznab -e 'select name from groups where active=1;' > groups.txt

You can now go into the groups.txt file and remove and additional groups you want or just leave it intact if you want everything.

Finally we want to run the php backfill.php command to backfill each of our selected groups back 365 days (or whatever number you’ve selected, alternatively you can also skip updating the database with the amount of days to backfill and simply substitute the backfill_date.php as noted from the link.)

Normally this command would be run like so for a single group:

php backfill.php alt.binaries.groupname.here

But this is much too slow for multiple groups so we are going to use a ‘for’ function with our data from the groups.txt file we created.

for group in `cat groups.txt`; do php backfill.php $group; done

If you are like me and like to see your code before it runs then simply add an echo beforehand to view the results and then remove it when you are ready:

for group in `cat groups.txt`; do echo "php backfill.php $group"; done

The use of the backticks (`) around ‘cat groups.txt’ tells it to run the command in a subshell and then use the information from that back in the shell command.