Shadow wiki - Automatically update a shadow copy of these wiki pages
Synopsis
/bin/bash shadow_wiki.sh
Remarks
WARNING: This page is for GridLAB-D administrators on SourceForge only. You know who you are. If you do not have admin privileges to the GridLAB-D project on SourceForge, then none of this will work for you.
You may create a shadow server for this wiki using the following bash script. This script synchronizes a local mediawiki server with the SourceForge wiki server. It should be installed in the htdocs folder of your web server.
#!/bin/bash # @file shadow_wiki.sh # @remark Save shadow_wiki.sh file in the htdocs folder of your local Apache server # to keep the local mediawiki in sync with the SourceForge mediawiki. rsync -avP -e ssh your-sf-id,gridlab-d@web.sourceforge.net:htdocs/wiki/ wiki wget http://gridlab-d.sourceforge.net/backup/mysqldumpz.php -O - -q | zcat | mysql --protocol=tcp --user=your-mysql-user --password=your-mysql-password --host=your-mysql-host --force
It will populate the wiki folder in htdocs. In this case you will access your server using the link http://localhost/wiki If this page is garbled (this does happen sometimes--see Bugs), then you can try logging in first.
Requirements
- You must have access to the SourceForge shell service with certificate-based authentication (see SourceForge SSH authentication for details).
- There must be a MySQL server running on a host named mysql-g (which can also by the localhost) with a user username using password password for ALL access to the schema g233096_mediawiki. The server should use the TCP protocol (do not use a sockets unless your server is set up for that, in which case you should change the --protocol=tcp option in the script). Authenticated GridLAB-D project administrators may obtain the username and password values at the SourceForge MySQL Database Administrator Information page.
- You must have rsync installed on your local host (see SourceForge rsync documentation for details).
- You must have wget installed on your local host.
- You must have mysql client installed on your local host.
- You must have a web server (e.g., apache, apache2) installed and started on your local host.
First time setup
To set up a shadow wiki for the first time, you should do the following (assuming you have setup your system correctly).
Manual download
First use the rsync command to obtain a copy of the GridLAB-D wiki tree on SourceForge:
/bin/bash% cd my-htdocs-foders /bin/bash% rsync -avP -e ssh your-sf-id,gridlab-d@web.sourceforge.net:htdocs/wiki/ wiki
Then retrieve a copy of the MySQL database dump for mediawiki and load it onto your system
/bin/bash% wget http://gridlab-d.sourceforge.net/backup/mysqldumpz.php -O gridlabd-mediawiki.sql.Z /bin/bash% zcat gridlabd-mediawiki.sql.Z | mysql --protocol=tcp --user=sf-gridlabd-mysql-user --password=sf-gridlabd-mysql-password --host=mysql-g
If you get any errors in this process, you must fix them before proceeding with fixing wiki/LocalSettings.php below.
Site name
You can distinguish your copy of the wiki page from the master copy by changing the following setting in wiki/LocalSettings.php:
$wgSitename = "GridLAB-D Wiki (Shadow Copy)";
Database Permissions
You can edit the wiki/LocalSettings.php to adjust the following settings:
$wgServer = "http://localhost"; $wgDBserver = "localhost"; $wgDBuser = "username"; $wgDBpassword = "password";
Read-only
You can prevent users from making edits to the local copy by changing the following settings in wiki/LocalSettings.php:
$wgGroupPermissions['user']['edit'] = false; $wgGroupPermissions['admin']['edit'] = false;
Protecting wiki/LocalSettings.php
If (and you really should) make changes to wiki/LocalSettings.php, then you should protect the file from future rsync updates by excluding it from the files rsync synchronizes. Do this by adding the --exclude wiki/LocalSettings.php command line option to the rsync command in the script.
Strong shadowing
It may be a good idea to keep the local wiki folders absolutely identical by deleting files that are deleted on the server. Do this by adding the --delete command line option to the rsync command in the script.
Email support
Even if you have email support enabled on your system, you should turn off all email processes by changing the following settings in wiki/LocalSettings.php:
$wgEnableEmail = false; $wgEnableUserEmail = false; $wgEmailAuthentication = false;
Download size
The server script backup/mysqldumpz.php delivers a compressed stream using gzip. The typical size of the script is about 20 MB compressed (which is about 100 MB uncompressed).
Making the script more robust
You can make the script more robust by chaining the commands together using && so that subsequent commands are not run if there is a failure, and adding traps for signals. An example of a more robust script is
. $0-conf cd $(dirname $0) test -f shadow_wiki.lck && test ! -z "$(ps $(cat shadow_wiki.lck) | tail --lines=+2 | cut -c1-5)" && exit echo $$ > shadow_wiki.lck function unlock { rm -if shadow_wiki.lck exit } trap unlock SIGINT SIGTERM function cleanup { rm -if gridlabd-mediawiki.sql.Z shadow_wiki.lck exit } trap cleanup EXIT ( rsync -avPq --delete -e "ssh -i $IDFILE" $SFUSER,gridlab-d@web.sourceforge.net:htdocs/wiki/ wiki --exclude LocalSettings.php && \ wget http://gridlab-d.sourceforge.net/backup/mysqldumpz.php -O gridlabd-mediawiki.sql.Z -q && \ zcat gridlabd-mediawiki.sql.Z | mysql --protocol=tcp --user=$MYUSER --password=$MYPWD --host=$MYHOST --force ) 1>shadow_wiki.log 2>&1
where the configuration file shadow_wiki.sh-conf is used
PATH=path-needed-to-find-tools-used IDFILE=your-rsa-id-file SFUSER=your-sf-login-id MYUSER=gridlabd-mysql-rw-user MYPWD=gridlabd-mysql-password MYHOST=gridlabd-mysql-host
Automatic Updates
You can use scheduled services to automatically update the wiki folder and wiki database. The specifics of doing this will depend on the platform that hosts your web server.
Linux
You may add the shadow_wiki.sh script to crontab for Apache. The command to edit Apache's crontab is
/bin/bash% sudo crontab -e -u apache
The line you should add
random-integer-from-0-to-59 * * * * your-htdocs-folder/shadow_wiki.sh
The reason you must have Apache update the files is that MediaWiki requires its files to be owned by the web server. However be aware the ssh_agent may prevent this from working correctly unless you set the ownership of the identity file to apache and permission to 600.
You may also wish to append a system log command at the end of the script, e.g.,
test -s shadow_wiki.log && logger -t shadow_wiki.sh "non-trivial log result (see $PWD/shadow_wiki.log for details)"
Mac OSX
You may add the shadow_wiki.sh script to launchctl using the load command. The launch control file shadow_wiki.plist should look like this:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>org.gridlabd.shadow_wiki</string> <key>ProgramArguments</key> <array> <string>path-to-htdocs-folder/shadow_wiki.sh</string> </array> <key>Nice</key> <integer>1</integer> <key>StartInterval</key> <integer>3600</integer> <!-- please, not more than once per hour --> <key>RunAtLoad</key> <true/> </dict> </plist>
In this case you may wish to append a system log command at the end of the robust script:
test -s shadow_wiki.log && syslog -s -l 3 "shadow_wiki.sh: non-trivial log result (see $PWD/shadow_wiki.log for details)"
Security
You must configure your web server to deny access to the shadow_wiki.sh-conf file and .ssh folder. For example on apache a virtual host configuration might be
<VirtualHost *:80> DocumentRoot your-htdocs-folder ServerName your-virtualhost-name ErrorLog /var/log/httpd/gridlabd-error-log CustomLog /var/log/httpd/gridlabd-access-log common </VirtualHost> <Directory "your-htdocs-folder"> Options Indexes FollowSymLinks ExecCGI Allow from all </Directory> <Files ~ "^shadow_wiki.sh-conf"> Order allow,deny Deny from all Satisfy All </Files> <Directory "your-htdocs-folder/.ssh"> Order allow,deny Deny from all Satisfy All </Directory>
Master/Proxy server
You may set up your system to act as a master or proxy server by downloading the backup folder from the master server:
rsync -avP your-sf-login-id,gridlab-d@web.sourceforge.net:htdocs/backup/ backup --exclude gridlabd-mediawiki\*
This will download all the files needed to support backups and synchronization, include the mysqldumpz.php needed to support wget access to the database dump used by the shadow_wiki.sh script.
If you get an error such as sh: mysqldump command not found
, you may have to add the following line to the second line of the script:
putenv("PATH=/bin:/usr/bin:/usr/local/bin:/usr/local/mysql/bin");
where the specific folder may need to be adjusted for your system.
Caveats
Prohibited rsync Usage
Absolutely do not ever use the --remove-source-files command line option in your rsync script! Doing so may destroy the entire mediawiki tree on SourceForge and all images, math equations, and uploaded files on the master server. This will inevitably come to the attention of the GridLAB-D project manager whose ire we do not wish to pique. As an administrator you should know better. But just in case, don't say we didn't warn you.
Bugs
Local ssh-agent
The local ssh-agent must be running for the login id that is executing the script if it is to access the identity file. This means that user who owns the identity file (e.g., apache) must be the user who runs the script. On many system, cron does not run "inside" an ssh-agent-enabled login shell and therefore cannot access the identity file. A solution to this problem has yet to be determined.
Character set encoding
There is a known problem with the character set encoding for some MySQL servers when not logged into the wiki. The solution is to log in using the Special:UserLogin page.
Timezone coordination
There are known problems with the MySQL server TIME_ZONE variable which cause the cache timestamps to be incorrect. This sometimes results in delays in updates being displayed, garbled data and missing page elements. The cause appears to the difference between SourceForge servers operating in UTC and the shadow server operating in the local time zone. The only known fix seems to be setting the shadow server in UTC also.
Special pages crash
The Special pages link crashes some servers when no user is logged in.