Layer 8 Consulting

Network Design and Computer Consulting

SOAP with SSL and Load Balancers

Sep 1

Posted by Brad in Uncategorized | No Comments

SOAP Stinks. There. I said it. The designers were really not paying attention when designing the whole wsdl thing.

It has a nasty habit of exposing the back end server IP addresses and forgetting that it's supposed to be SSL.

I had lousy luck making Pound, HAProxy, UltraMonkey, and NGINX play nice as load balancers, but finally got Apache to work. Now that I got it, the others might work using the same techniques, NGINX is probably the closest to working.

Details: https://public.address.com/ebfUploader.asmx?wsdl or ?DISCO
returns the improper response: <soap:address location="http://10.100.111.40/ebfUploader.asmx"/>
because the server is unaware that the site is being proxied into SSL on another server.

Here's how I made it work:

- Use Apache 2.2 as the load balancer
- **MUST HAVE** valid certificate for the named site you are deploying to! Self signed certs do not work.
- If you visit your site (i.e.: https://public.address.com/ebfuploader.asmx?wsdl) in Internet Explorer, it
MUST NOT give any complaints about the certificate!
- The backend servers are normal http, and state server or cookie backed for session tracking
- The load balancer config must dynamically alter the text/html and replace the http private addresses with https public addresses:
Sample config for Apache:

apacheproxy:/etc/apache2/sites-available# cat lbtest-ssl
<IfModule mod_ssl.c>
<VirtualHost _default_:443>
ServerAdmin webmaster@localhost
ProxyPass /balancer-manager !
<Proxy balancer://mycluster>
BalancerMember http://10.100.110.101:80/
BalancerMember http://10.100.110.102:80/
BalancerMember http://10.100.110.103:80/
BalancerMember http://10.100.110.104:80/
#Failed everything, give up, go to lastmanstanding server
BalancerMember http://10.100.110.168:8080/ status=+H
ProxySet lbmethod=bytraffic
Allow from all
</Proxy>
ProxyPass / balancer://mycluster/
ProxyPassReverse / balancer://mycluster/
AddOutputFilterByType SUBSTITUTE text/xml
Substitute "s|http://10.100.110.101|https://public.address.com|in"
Substitute "s|http://10.100.110.102|https://public.address.com|in"
Substitute "s|http://10.100.110.103|https://public.address.com|in"
Substitute "s|http://10.100.110.104|https://public.address.com|in"

SSLEngine on
SSLCertificateFile    /etc/ssl/certs/star.address.com.pem
SSLCertificateKeyFile /etc/ssl/private/star.address.com.key

</VirtualHost>
</IfModule>

ZRM for MySQL issues with big blob in database

Aug 20

Posted by Brad in backup, linux, mysql | No Comments

I'm still on the fence with using ZRM/Amanda over my usual db dump scripts and backuppc, but just in case, here's a gotcha with mysqldump's in general. The granularity of restores being my biggest issue, although maybe I'm just doing it wrong. I'll keep looking.

ZRM gives errors dumping my db's with big blobs in them, typically we just add a --max_allowed_packet=1024M switch into the mysqldump command line. The trouble with ZRM is that it builds the commands on the fly with perl, and it looked ugly to just kludge that into there somehow. Turns out, you can add it to the backup server's /etc/mysql/my.cnf file in the [mysqladmin] section. Add a new line with: max_allowed_packet=1024M and save it. From that point on, that switch is honored when the mysqldump runs.

Problem solved.

Building a slow server

Aug 6

Posted by Brad in Uncategorized | No Comments

How many times do you have to build a slow server on purpose? Weird. But useful for testing what your web/vpn app will behave like from the other side of the country. It's easier than it used to be!
Build yourself a regular Debian server with whatever you want to test on it.
Copy this into a script called 512kbps.sh:
tc qdisc del dev eth0 root
tc qdisc add dev eth0 root handle 1: tbf rate 512kbit buffer 1600 limit 3000
tc qdisc add dev eth0 parent 1: handle 10: netem delay 150ms

Copy this into a script called resetwan.sh:
tc qdisc del dev eth0 root

Make them executable with chmod +x *.sh

Run ./512kbps.sh and presto! your server acts like it's at the other end of a 512kbps, 150ms internet link!
Run ./reset.sh to get it back to normal.

Use the 512kbps.sh as your template to set up multiple speeds and ping times, the script is pretty easy to figure out where the numbers go.

Slow down, and take it easy!

Jailbreak your 4.0 iPhone!

Aug 1

Posted by Brad in howto, iphone | No Comments

Go to Jailbreakme.com and slide the slider. Sanity returns, and cydia is added to the program list. Run cydia, choose user, search for "categories", click install, and Brad is a happy camper again. Sweet!

Tags:

Adito won't restart, Failed to start Jetty. Multiple exceptions

Jul 23

Posted by Brad in Uncategorized | No Comments

Short answer: /opt/adito-0.9.1/conf/webserver.properties is hosed. The GUI can screw it up and leave multiple \!\! in the file where there should only be \!
Edit the webserver.properties file and search and replace the \!\! with \! and restart adito, it should be fine now.

Tags:

iPhone iOS 4.0 is stupid

Jul 4

Posted by Brad in iphone | No Comments

Yikes. Where to begin.
Lost 3G data connectivity for a week. Apple couldn't fix. AT&T couldn't fix. Had to start over from factory defaults. Awesome. Lost all SMS history and call history, 4 email accounts to set up again, facebook, woot, ssh connections, rdp connections, vnc connections, VPN, etc. All gone. Every app is starting over, no settings, no ring tones, no songs, no podcasts, no bookmarks. Ridiculous.
Categories? Apple, do you even look at the JB apps before you release something? 12 apps per folder and no icons for them? Pure genius. Now instead of 6 folders and 2 screens of apps, one a daily-use easy peasy screen and one of aptly named and iconed folders that my four year old can use, I have 11 pages of apps and no hope of ending up with anything more than 4 pages of identical folders distinguished by ittybitty text that no one can tell apart at all and even then, is foursquare in social networking 3 or search apps 2? Who knows? I'm reduced to using finder for everything short of the phone and iPod. Stupid stupid stupid! Had it grind to a halt and just die last night. Had a mystery app eat the battery today in about 30 minutes and refuse to charge until it rebooted again. Tethering but only over USB or Bluetooth? Have you seen MyWi?? iTunes app management is still for meth addicts. No one else should have patience to drag things around into any semblance of sanity. It shows categories that kinda make sense, but are unusable anywhere.
Roll back to 3.1.3? Hell yes, if I can figure out what mystical incantations are required to get it to reinstall! Apple is apparently smoking the Microsoft weed again.

Steve Jobs: how many apps are on your phone? Really? Who approved "12 to a group"? Idiots.

Tags:

Endian firewalls stop emailing the backups to you

Jun 22

Posted by Brad in backup, endian, firewall | 1 Comment

The crontab for root gets pooched somewhere along the line.

SSH onto the firewall, cd to /var/spool/cron

You'll have  files named root, root.orig, root.bak, and root.bak.orig.

If you cat the root file, it's clearly hosed up. One of the others will have a normal cron setup like:

SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=
HOME=/

#
&nolog */1 * * * *      [ -x /bin/run-parts ] && run-parts --report /etc/cron.minutely
&nolog */5 * * * *      [ -x /bin/run-parts ] && run-parts --report /etc/cron.cyclic
01  * * * *     [ -x /bin/run-parts ] && run-parts --report /etc/cron.hourly
25  1 * * *     [ -x /bin/run-parts ] && run-parts --report /etc/cron.daily
47  2 * * 0     [ -x /bin/run-parts ] && run-parts --report /etc/cron.weekly
52  3 1 * *     [ -x /bin/run-parts ] && run-parts --report /etc/cron.monthly

#
@nolog 5        [ -x /bin/run-parts ] && run-parts --report /etc/anacron.cyclic
@ 1h    [ -x /bin/run-parts ] && run-parts --report /etc/anacron.hourly
@ 1d    [ -x /bin/run-parts ] && run-parts --report /etc/anacron.daily
@ 1w    [ -x /bin/run-parts ] && run-parts --report /etc/anacron.weekly
@ 1m    [ -x /bin/run-parts ] && run-parts --report /etc/anacron.monthly

# Summarize ip accounting info:
2 1 * * *       /usr/sbin/ipacsum -r -H `/bin/hostname` -t "the day 2 days ago" >/dev/null
3 1 * * 0       /usr/sbin/ipacsum -r -H `/bin/hostname` -t "the week 4 weeks ago" >/dev/null
4 1 1 2 *       /usr/sbin/ipacsum -r -H `/bin/hostname` -t "the year 2 years ago" >/dev/null
57  23 * * *     [ -x /usr/lib/cron/logrotate.cron ] && /usr/lib/cron/logrotate.cron

So, delete the root file, cp root.orig root, and off you go!

Fix for poor throughput on Windows 2003 Xen Guest

Jun 22

Posted by Brad in performance, windows, xen | No Comments

I had a box that would work for the most part, but rsync backups would take HOURS for a single 90MB zip file. That's not right. After fighting with cygwin dll versions and updated cwrsync versions, it turns out that it is a known issue with TCP checksum offloading. Apply this registry change, reboot the guest, and ZING! you're off!

ADD: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameter\DisableTaskOffload as a REG_DWORD, then edit it so that it equals 1

Reboot the guest.

Citrix XenServer Windows console stuck in 4bit color

May 19

Posted by Brad in windows, xen | No Comments

Windows guests on XenServer can occasionally lose their video config on the Xen console and get stuck in 640x480 4bit color. Changing it the normal way doesn't actually change anything.
Here's the fix.
Open device manager and delete the Cirrus 4mb video card and reboot the virtual machine.

That allows it to redetect the video and come back normally. Rdp sessions are unaffected, but xvp and vnc connections are.

Ubuntu Server 10.04 ureadahead problems

May 12

Posted by Brad in debian, linux, ubuntu, xen | No Comments

The newest Ubuntu Server is a little disappointing. I built a 9.10 server that boots in like four seconds, so it's funny that the problems I have with 10.04 center around the ureadahead daemon whose whole purpose is to speed up the boot process.

I built one with software mirrored disks and one RAID 5 set. After updating with an aptitude update && aptitude upgrade, it seemed to be running ok and rebooted ok. I added the RAID 5 to the fstab and to the /etc/exports for the NFS share I needed. I rebooted it and HANG! Drat.

Google coughs up a lot of people having similar issues with ureadahead giving a legitimate error, but freaking out init to the point where it won't boot any further. The developer of it pipes up and explains what it does and why it does etc. etc. here. You know what? It's still hanging my new NFS box, and that's not good.

You can disable readahead by running these two lines:

sudo mv /etc/init/ureadahead.conf /etc/init/ureadahead.conf.disable

sudo mv /etc/init/ureadahead-other.conf /etc/init/ureadahead-other.conf.disable

How often do you reboot your server anyway? It still blows the doors off the Win 2003R2 box next to it.

That was not the end of it though, more problems with fsck hanging on startup and then the gigabit card wouldn't work either, the ns83820 driver for it kept saying that the MAC address was fffffffffffffff.

I know it worked the day before with FreeNAS. Heck with it. Debian 5.04 Server installed, and it all works fine.

Debian works far more easily on XenServer than Ubuntu, so I'm sticking with Debian for servers for now. Ubuntu 10.04 Workstation on my EEEPc is lightning fast! Boots in 13 seconds, shuts down in 6. I'm pretty sure it can reboot faster than my iPhone.

Bad Behavior has blocked 17 access attempts in the last 7 days.