Behind the scenes at CodeNewsFast

www.codenewsfast.com is the current web address of our searchable archive of Delphi support posts, now with more than 17gb of data. The archive was called developers.href.com in the 1990s. It was purely based on NNTP content for more than 10 years. NNTP was great and almost completely free to use and run but it was not slick and exciting and development talent did not show up to make it robust enough. We tried for years to get by with an open-source nntp solution on CentOS linux but it really did not hold up.

It took quite a while for me to decide that Discourse was popular enough to give it some extra attention.

The Australian Delphi User Group has given us permission to archive their content, so we are working on a custom import from its RSS and JSON URLs. Once that is working, we’ll try to bring in some of the other Delphi discourse forums.

There are still about 5 active NNTP Delphi forums. These days we have a multi-step process which involves using XanaNews for downloading from each of those newsgroups. We import from there into Firebird SQL, thread the conversations and re-index. I always say that I will do it every 3 months, hmm, the last import was more than a year ago. I ran into some trouble with a deadlock and have been trying to fix the logic around it.

1 Like

Hi Ann,

I am wondering what level of automation do you have around the quarterly NNTP import?

Thanks,
Ronan

There is less and less NNTP import, but just to answer your question, we have human-clicking within XanaNews to get the messages, not difficult, then a series of delphi programs that run in order to massage the data to the format which gets imported, then imported, and then re-indexed by Rubicon.

Right now we are working on a new source: discourse forum data, which is available via JSON and RSS for most forums.

The transfer format is based on the NNTP standard, in plain-text, with a range of articles included in one .news file.

Ann

Hi Ann,

Thanks for the details. CodeNewFast is a - great service - I have used the countless times over the years.

Best regards,
Ronan