From GridLAB-D Wiki
Jump to: navigation, search

Shadow wiki - Automatically update a shadow copy of these wiki pages

Synopsis

/bin/bash shadow_wiki.sh 

Remarks

WARNING: This page is for GridLAB-D administrators on SourceForge only. You know who you are. If you do not have admin privileges to the GridLAB-D project on SourceForge, then none of this will work for you.

You may create a shadow server for this wiki using the following bash script. This script synchronizes a local mediawiki server with the SourceForge wiki server. It should be installed in the htdocs folder of your web server.

#!/bin/bash
# @file   shadow_wiki.sh
# @remark Save shadow_wiki.sh file in the htdocs folder of your local Apache server
#         to keep the local mediawiki in sync with the SourceForge mediawiki. 
rsync -avP -e ssh your-sf-id,gridlab-d@web.sourceforge.net:htdocs/wiki/ wiki
wget http://gridlab-d.sourceforge.net/backup/mysqldumpz.php -O - -q | zcat | mysql --protocol=tcp --user=your-mysql-user --password=your-mysql-password --host=your-mysql-host --force

It will populate the wiki folder in htdocs. In this case you will access your server using the link http://localhost/wiki If this page is garbled (this does happen sometimes--see Bugs), then you can try logging in first.

Requirements

  1. You must have access to the SourceForge shell service with certificate-based authentication (see SourceForge SSH authentication for details).
  2. There must be a MySQL server running on a host named mysql-g (which can also by the localhost) with a user username using password password for ALL access to the schema g233096_mediawiki. The server should use the TCP protocol (do not use a sockets unless your server is set up for that, in which case you should change the --protocol=tcp option in the script). Authenticated GridLAB-D project administrators may obtain the username and password values at the SourceForge MySQL Database Administrator Information page.
  3. You must have rsync installed on your local host (see SourceForge rsync documentation for details).
  4. You must have wget installed on your local host.
  5. You must have mysql client installed on your local host.
  6. You must have a web server (e.g., apache, apache2) installed and started on your local host.

First time setup

To set up a shadow wiki for the first time, you should do the following (assuming you have setup your system correctly).

Manual download

First use the rsync command to obtain a copy of the GridLAB-D wiki tree on SourceForge:

/bin/bash% cd my-htdocs-foders
/bin/bash% rsync -avP -e ssh your-sf-id,gridlab-d@web.sourceforge.net:htdocs/wiki/ wiki

Then retrieve a copy of the MySQL database dump for mediawiki and load it onto your system

/bin/bash% wget http://gridlab-d.sourceforge.net/backup/mysqldumpz.php -O gridlabd-mediawiki.sql.Z
/bin/bash% zcat gridlabd-mediawiki.sql.Z | mysql --protocol=tcp --user=sf-gridlabd-mysql-user --password=sf-gridlabd-mysql-password --host=mysql-g

If you get any errors in this process, you must fix them before proceeding with fixing wiki/LocalSettings.php below.

Site name

You can distinguish your copy of the wiki page from the master copy by changing the following setting in wiki/LocalSettings.php:

$wgSitename = "GridLAB-D Wiki (Shadow Copy)";

Database Permissions

You can edit the wiki/LocalSettings.php to adjust the following settings:

$wgServer = "http://localhost";
$wgDBserver = "localhost";
$wgDBuser = "username";
$wgDBpassword = "password";

Read-only

You can prevent users from making edits to the local copy by changing the following settings in wiki/LocalSettings.php:

$wgGroupPermissions['user']['edit'] = false;
$wgGroupPermissions['admin']['edit'] = false;

Protecting wiki/LocalSettings.php

If (and you really should) make changes to wiki/LocalSettings.php, then you should protect the file from future rsync updates by excluding it from the files rsync synchronizes. Do this by adding the --exclude wiki/LocalSettings.php command line option to the rsync command in the script.

Strong shadowing

It may be a good idea to keep the local wiki folders absolutely identical by deleting files that are deleted on the server. Do this by adding the --delete command line option to the rsync command in the script.

Email support

Even if you have email support enabled on your system, you should turn off all email processes by changing the following settings in wiki/LocalSettings.php:

$wgEnableEmail = false;
$wgEnableUserEmail = false;
$wgEmailAuthentication = false;

Download size

The server script backup/mysqldumpz.php delivers a compressed stream using gzip. The typical size of the script is about 20 MB compressed (which is about 100 MB uncompressed).

Making the script more robust

You can make the script more robust by chaining the commands together using && so that subsequent commands are not run if there is a failure, and adding traps for signals. An example of a more robust script is

. $0-conf
cd $(dirname $0)
test -f shadow_wiki.lck && test ! -z "$(ps $(cat shadow_wiki.lck) | tail --lines=+2 | cut -c1-5)" && exit
echo $$ > shadow_wiki.lck
function unlock {
        rm -if shadow_wiki.lck
        exit
}
trap unlock SIGINT SIGTERM
function cleanup {
        rm -if gridlabd-mediawiki.sql.Z shadow_wiki.lck
        exit
}
trap cleanup EXIT
( rsync -avPq --delete -e "ssh -i $IDFILE" $SFUSER,gridlab-d@web.sourceforge.net:htdocs/wiki/ wiki --exclude LocalSettings.php && \
  wget http://gridlab-d.sourceforge.net/backup/mysqldumpz.php -O gridlabd-mediawiki.sql.Z -q && \
  zcat gridlabd-mediawiki.sql.Z | mysql --protocol=tcp --user=$MYUSER --password=$MYPWD --host=$MYHOST --force ) 1>shadow_wiki.log 2>&1

where the configuration file shadow_wiki.sh-conf is used

PATH=path-needed-to-find-tools-used
IDFILE=your-rsa-id-file
SFUSER=your-sf-login-id
MYUSER=gridlabd-mysql-rw-user
MYPWD=gridlabd-mysql-password
MYHOST=gridlabd-mysql-host

Automatic Updates

You can use scheduled services to automatically update the wiki folder and wiki database. The specifics of doing this will depend on the platform that hosts your web server.

Linux

You may add the shadow_wiki.sh script to crontab for Apache. The command to edit Apache's crontab is

/bin/bash% sudo crontab -e -u apache

The line you should add

random-integer-from-0-to-59 * * * * your-htdocs-folder/shadow_wiki.sh

The reason you must have Apache update the files is that MediaWiki requires its files to be owned by the web server. However be aware the ssh_agent may prevent this from working correctly unless you set the ownership of the identity file to apache and permission to 600.

You may also wish to append a system log command at the end of the script, e.g.,

test -s shadow_wiki.log && logger -t shadow_wiki.sh "non-trivial log result (see $PWD/shadow_wiki.log for details)"

Mac OSX

You may add the shadow_wiki.sh script to launchctl using the load command. The launch control file shadow_wiki.plist should look like this:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
 <key>Label</key>
 <string>org.gridlabd.shadow_wiki</string>

 <key>ProgramArguments</key>
 <array>
   <string>path-to-htdocs-folder/shadow_wiki.sh</string>
 </array>

 <key>Nice</key>
 <integer>1</integer>

 <key>StartInterval</key>
 <integer>3600</integer> <!-- please, not more than once per hour -->

 <key>RunAtLoad</key>
 <true/>
</dict>
</plist>

In this case you may wish to append a system log command at the end of the robust script:

test -s shadow_wiki.log && syslog -s -l 3 "shadow_wiki.sh: non-trivial log result (see $PWD/shadow_wiki.log for details)"

Security

You must configure your web server to deny access to the shadow_wiki.sh-conf file and .ssh folder. For example on apache a virtual host configuration might be

<VirtualHost *:80>
       DocumentRoot your-htdocs-folder
       ServerName your-virtualhost-name
       ErrorLog /var/log/httpd/gridlabd-error-log
       CustomLog /var/log/httpd/gridlabd-access-log common
</VirtualHost>
<Directory "your-htdocs-folder">
       Options Indexes FollowSymLinks ExecCGI
       Allow from all
</Directory>
<Files ~ "^shadow_wiki.sh-conf">
       Order allow,deny
       Deny from all
       Satisfy All
</Files>
<Directory "your-htdocs-folder/.ssh">
       Order allow,deny
       Deny from all
       Satisfy All
</Directory>

Master/Proxy server

You may set up your system to act as a master or proxy server by downloading the backup folder from the master server:

rsync -avP your-sf-login-id,gridlab-d@web.sourceforge.net:htdocs/backup/ backup --exclude gridlabd-mediawiki\*

This will download all the files needed to support backups and synchronization, include the mysqldumpz.php needed to support wget access to the database dump used by the shadow_wiki.sh script.

If you get an error such as sh: mysqldump command not found, you may have to add the following line to the second line of the script:

putenv("PATH=/bin:/usr/bin:/usr/local/bin:/usr/local/mysql/bin");

where the specific folder may need to be adjusted for your system.

Caveats

Prohibited rsync Usage

Absolutely do not ever use the --remove-source-files command line option in your rsync script! Doing so may destroy the entire mediawiki tree on SourceForge and all images, math equations, and uploaded files on the master server. This will inevitably come to the attention of the GridLAB-D project manager whose ire we do not wish to pique. As an administrator you should know better. But just in case, don't say we didn't warn you.

Bugs

Local ssh-agent

The local ssh-agent must be running for the login id that is executing the script if it is to access the identity file. This means that user who owns the identity file (e.g., apache) must be the user who runs the script. On many system, cron does not run "inside" an ssh-agent-enabled login shell and therefore cannot access the identity file. A solution to this problem has yet to be determined.

Character set encoding

There is a known problem with the character set encoding for some MySQL servers when not logged into the wiki. The solution is to log in using the Special:UserLogin page.

Timezone coordination

There are known problems with the MySQL server TIME_ZONE variable which cause the cache timestamps to be incorrect. This sometimes results in delays in updates being displayed, garbled data and missing page elements. The cause appears to the difference between SourceForge servers operating in UTC and the shadow server operating in the local time zone. The only known fix seems to be setting the shadow server in UTC also.

Special pages crash

The Special pages link crashes some servers when no user is logged in.

See also