opencaching.com

hmarq · Site Admin Joined: 15 Sep 2003 Posts: 351

OK, so the concensus is to use nntp for replicating ... at least as an initial premise.

Does anyone have a favorite nntp server, so we can start talking about configuring and building an experiemental backbone? ... seems like it would be a good idea to all use the same type of server so that config issues could be discussed on a level field.

Steve · Joined: 17 Sep 2003 Posts: 21 Location: Thousand Oaks, CA

I've never set up a NNTP server, but I believe that INN is what comes standard with RedHat, which is likely to be the most common distro/os used.

It shouldn't matter which one is used, as long as it has all the features we're going to require and it's RFC compliant, so other NNTP servers can talk to it properly.
_________________
Steve

Vinnie · Joined: 23 Sep 2003 Posts: 71 Location: Cologne, Germany

As far as I know, "INN" is the choice of all major NNTP sites.

License is BSD.
Web-Site is http://www.isc.org/products/INN/

I once compiled and set up INN on digital unix, and it was not too complicated to get it up and running and serving as a "team news server". If you use a modern linux distribution, it is of course even easier: install the RPM or DEB and you have a running news server.

There are many other small and simple nntp servers out there, but I prefer the good old solid INN, even if it requires an hour or two more to learn how to handle it.

amnesius · Joined: 27 Sep 2003 Posts: 147 Location: ./earth/europe/*

Hi folks,

well im reading about the nntp server issue everywhere in the forum but i cant really understand what for you want to use it?

*Confused*

Can anyone help me out?

Cheers

-Stefan
_________________
~Amnesius

amnesius · Joined: 27 Sep 2003 Posts: 147 Location: ./earth/europe/*

Ok, i got it now.. You guys want to use NNTP as a communication protocol between the nodes, right?

Well for me its not sure if this is a good way to handle the replication. Dont you think a HTTP based communication will do it? You can exchange XML based node updates using simple http requests. I suggest this because on most of the chartered servers you cant install your own services..

Cheers

-Stefan
_________________
~Amnesius

Vinnie · Joined: 23 Sep 2003 Posts: 71 Location: Cologne, Germany

amnesius · Joined: 27 Sep 2003 Posts: 147 Location: ./earth/europe/*

Well i got it so far.. but can you install an nntp server on a chatered server?

Cheers

-Stefan
_________________
~Amnesius

raven · Joined: 29 Sep 2003 Posts: 84 Location: Bielefeld, Germany

if you mean an ordinary webspace/hosting provider by "chatered" then, no, you can't install an nntp server there. but you probably don't want to run a core node anyway, if you have just some webspace at your hosting provider.
don't get me wrong, you don't need to run a full-blown core-node if you just want (read) access to the cache data for something like a stats site. being a core-node is only neccessary if you want to run a something like a cache-listing service like gc.com or navicache, where you need ALL the data and you need write access to it. and if you want to do something like that, you should probably consider getting your own server (at work, at home, co-located own machine or a rented root-server or something like that) anyway.
_________________
Quoth the raven, "Nevermore"

amnesius · Joined: 27 Sep 2003 Posts: 147 Location: ./earth/europe/*

@Raven: Ok, i got it..

Concerning your replication strategy, have you ever thought about the lag when distributing the data?
_________________
~Amnesius

hmarq · Site Admin Joined: 15 Sep 2003 Posts: 351

amnesius · Joined: 27 Sep 2003 Posts: 147 Location: ./earth/europe/*

Hi,

after a good sleep and a coffe everything looks a bit different now. I agree that the lag doesnt really matter.

But some other things concerning NNTP came up to my mind.

First of all AFAIK NNTP does not support deleting or altering an article and distributing this to other nodes.

Well you might say that you will only forward change messages which will be processed by the nodes. But this means you have to make a huge load of those messages available to keep the nodes in sync after an autage.

Another issue how can you initialize a new node? Do you think about letting this node fetch all change messages ever been sent?

Maybe you have a solution for this already, this is what came to my mind when i thought about it again by this morning..

Cheers

-Stefan
_________________
~Amnesius

raven · Joined: 29 Sep 2003 Posts: 84 Location: Bielefeld, Germany

amnesius · Joined: 27 Sep 2003 Posts: 147 Location: ./earth/europe/*

raven · Joined: 29 Sep 2003 Posts: 84 Location: Bielefeld, Germany

umm, yes, diskspace surely is an issue in this whole thing, but not specifically in the replication process. and looking at today's harddisk-prices and the kind of data we are going to manage here (mostly text-stuff) i don't think this issue is really that big.
take our webserver (hosting 9 domains + mails + db) for example, it still has lots of diskspace available, diskspace isn't really a big issue here, i think.
_________________
Quoth the raven, "Nevermore"

amnesius · Joined: 27 Sep 2003 Posts: 147 Location: ./earth/europe/*

Ok if you talk about simple hosting i agree with you 100%

But for this case you have a DB containing all data. And beside the DB you have an NNTP Server containing all the data + changes for a specified amount of time.

This means you have 2 datasources containing redundant data. From this datasources the NNTP Server size will grow alot faster then the datasize of the DB.

Ok diskspace is cheap, but think bigger after you reached a critical mass you need to think about disk performance for seeks and reorg's.

You might want to have a Raid0 Stripeset to increase the performance.

Uh and i forgot something really essencial, pics! What about pics? Do you think about spreading them over the NNTP feed too? If im not wrong binary grow when you encode whem bin2ascii ( not 100% sure ). If you do this you will have the picture twice on your node too..

Cheers

-Stefan
_________________
~Amnesius