Webcluster Interface for DXSpider

Hans-J.Barthen, DL5DI

jan 04th 2013

1. History:

I used the webcluster of N5IN during last years on my different webpages
but it is just crazy to have DX-spider running locally and get the 
webcluster from a server in the US :)

I did not find any webcluster software on the internet to run locally
with dx-spider (other than the Java-client of course).
So I spent a weekend creating this simple script.
It is just a quick hack and for sure there may be a lot of ideas for possible

The webpage is created by a perl-script, very easy, I am surely not an expert
in perl like Dirk Koopman.

2. Requirements and limitations:

- the script has only been tested on my LinuX-server yet.
  I am running 
    - Perl Version 5.8.8 
    - Apache 2.2.3 
    - OpenSuse 10.3
    - DX-Spider 1.54

- spider is running in standard paths and configuration

I do not expect any big issues on other distributions and versions but
in case of different paths for cgi-bin and local binaries or the 
spider database files the script needs to be adjusted to your needs.

- the webpage is not user-configurable like the N5IN webcluster, the
user can not set any filters or whatever.
This would make php or java necessary, so the user can better use the 
existing java-applet for dxspider.
I found in my logfiles that nearly nobody was using the filters of the
N5IN webcluster here.

- The script accepts only 1 parameter with the number of spots that 
should be shown in the first table. With this the size of the output
can be adjusted to fit best to your webpage, frame, window or wherever
you want to place it.

- the script is written to run on the same PC like dx-spider. 
It needs direct access to dxspider's filesystem.

3. How it works:

- the script reads the actual logfiles for DX-Spots, WWV-, WCY-datas and
ANNouncements of dxspider.
    - the number of WWV-, WCY- and ANN-lines is fixed to 3 each.
    - the number of spots is variable and can be given by a parameter.
      The default number of spots is 20, minimum is 1, maximum is 50.
      This values can be changed in the script.

- DXSpider changes the logfiles regular.
Depending on the actual traffic you may see that tables are quiet small
in the first view minutes after such a change of the logfiles.

This happens daily at 0:00 UTC with the dx-spots and at the start of a
new month with the WCY/WWV/ANN datas.
In Europe this is no big problem, 0:00 UTC is always in the night between
0:00 and 2:00 localtime, when only a few people are using the cluster.
At other places on the world this might be during main traffic.

I thought about reading the old logfiles if the new have not enough
content to fill the tables, but this is some bigger effort.
Just a few ideas what it could mean:
- the change of the log at the change of a day may be also a change of
the month or even of the year.
- an empty logile may also be caused by other reasons like
    - the cluster is not running,
    - no internet connection available or whatever,
so the last available datas may be totally outdated or not available at all.

4. Installation

The tgz contains the following files:

    a copy of the GNU General Public Licence

    this is a simple cgi-script, located in the cgi-bin-path of the
    In my case it is located at /var/local/httpd/cgi-bin/

    this file will be created automatically in the cgi-bin directory.
    It saves the value of the hit counter of the script.
    The value is shown in the right down corner of the webpage.

    this is the perl-script doing the job.
    In my case it is located in /usr/local/bin
    It is called from the cgi-script.
    Please adjust the path in the cgi-script if you locate it at a
    different place.

    Just 2 samples to adjust the number of spots to a fix value of 10
    or 35.
    Another possible way is to call the cgi-script with a parameter 
    like this:

    The 2 flags of my header at http://dx.db0j.de
    Take what you like to see in your header.
    This grafics are located in the main document root of the webserver.

    dxcc.list is a simple textfile with the DXCC-prefixes.
    It is needed to get the prefix of a country from the dxcc number
    in the database.
    dxcc.list is generated from the content of prefix_data.pl by the
    tool "convert-dxcc".
    prefix_data.pl is part of dxspider and located in the data directory.
    Place convert-dxcc whereever you like, I keep it in /usr/local/sbin.
    Start it whenever prefix_data.pl has been updated with dx-spider 
    dxcc.list is created in the data directory of dxspider.
    It may somehow be possible to use prefix_data.pl direct but I must
    admit that I don't know how??

5. Configuration and changes:

Changes need to be done direct in the code of the perl script.
You will find some helpfull comments in the script.

Things that possibly need to be changed:

- The header and the footer !
Please don't forget to take out my cluster's callsign.
This is no big problem, just some simple html-code in a perl-print-command.

An easy way to do some bigger changes:
Start webcluster.pl manually and let the output run to a file.
Like this: ./webcluster.pl > testfile.html

Take an html-editor and change the head and bottom of the file
testfile.html to your needs.

Now the critical part: Copy the changes from the html-source to the
print command lines of the perl script.
(Remember to escape special characters like ", ' with a \ !)

I created a table for the top with the name of the cluster and some
small grafics and another table for the footer with some links and the

- The automatic refresh interval of the page is set to 150sec.
You may change this parameter in the HTML-code of the header which is
generated by the perl-script. Just search for "refresh" and change the 
"CONTENT" parameter from 150 to your preferred value.

- The colors of the different tables?
This is done in the top of each table by HTML-code.

Just be carefull not to change the code which creates the tables with
the variable datas.

6. Possible issues and limitations:

- I tried to filter out all special characters that might come in with 
the text of a spot and destroy the format of the tables.
If you see any formatting issues please let me know!
Send me the sample of the spot and the exact time in UTC.

- The script automatically creates an URL for the spotted callsigns to
look it up at qrz.com just by a mouse-klick.
I think this is very helpfull for users.
Off course the script does not know or check if an entry at qrz.com
really exists, but most interesting dx-stations have it.
Sometimes it is not easy to find the basic callsign for this lookup 
with all additional prefixes, suffixes, extra short and extra long
special event callsigns etc.

The script expects that the basic callsign in general is the longest 
part of the string.
Based on this it checks if it contains at least 1 number and 2 chars.
Please give me a sample if you see a case where this does not work!

- I did not limit the length of announcements. Big story-tellers may
create more lines at this place than expected and the table will
grow bigger than planned.
If you want to prevent this please just cut announcements to the size
that you like.

Any idea for improvement is welcome.
Please either do it yourself if you can or tell me your idea and I
will try to implement it.
I would like to keep the most interesting versions on my server for
download, so if you do a nice improvement please send me a copy!

The URL is http://download.prgm.org/dl5di-soft/dxspider-webcluster
There you will find this files:

DXSpider-Webcluster.pdf         short documentation in PDF format
counter.txt                     information for counter setup
<old>                           former versions of the script
readme.txt                      this text
update-info.txt                 how to install the latest update
webcluster.20130103.tgz         latest version in tgz format
webcluster.20130103.zip         latest version in zip format
webcluster.patch20130103.tgz    update from former to latest version in tgz
webcluster.patch20130103.zip    update from former to latest version in zip
webcluster.tgz                  link to latest version (tgz)
webcluster.zip                  link to latest version (zip)
webcluster-patches.tgz		link to latest patches (tgz)
webcluster-patches.zip		link to latest patches (zip)

The patches should make sure that you keep your own settings.
Depending on your own setip it may show some differences in the line 
numbers where changes have been found.

The full version requires local adjustments!

The package is distributed under GNU/General Public Licence conditions.

If you find something better or more powerfull let me know,
I am interested!

Thanks !

Have fun and gd dx !

73 de Hans, DL5DI

Email: dl5di at gmx.de