Nathaniel and Sheree back in Australia

I really should have posted about this earlier but life has been a little bit hectic over the last couple of days. On the 13th of March 2009 Zander James Brown was born. Enjoy the picutres:

Easter Zander

Dad & Zander

Mum, Dad & Zander

This is a meme that is going around and it looks interesting. A lot of it is very American but that’s life. I first read it at Empty Thoughts.

1. Started your own blog

2. Slept under the stars

3. Played in a band

4. Visited Hawaii (Visited Vanuatu & Malaysia which are closer to Australia but similar)

5. Watched a meteor shower (Tried to 3 times but failed)

6. Given more than you can afford to charity

7. Been to Disneyland

8. Climbed a mountain

9. Held a praying mantis

10. Sang a solo

11. Bungee jumped

12. Visited Paris

13. Watched a lightning storm

14. Taught yourself an art from scratch

15. Adopted a child

16. Had food poisoning

17. Walked to the top of the Statue of Liberty

18. Grown your own vegetables

19. Seen the Mona Lisa in France

20. Slept on an overnight train

21. Had a pillow fight

22. Hitch hiked

23. Taken a sick day when you’re not ill

24. Built a snow fort

25. Held a lamb

26. Gone skinny dipping

27. Run a Marathon

28. Ridden in a gondola in Venice

29. Seen a total eclipse

30. Watched a sunrise or sunset

31. Hit a home run

32. Been on a cruise

33. Seen Niagara Falls in person

34. Visited the birthplace of your ancestors

35. Seen an Amish community

36. Taught yourself a new language (Mandarin)

37. Had enough money to be truly satisfied

38. Seen the Leaning Tower of Pisa in person

39. Gone rock climbing

40. Seen Michelangelo’s David

41. Sung karaoke

42. Seen Old Faithful geyser erupt

43. Bought a stranger a meal at a restaurant

44. Visited Africa

45. Walked on a beach by moonlight

46. Been transported in an ambulance

47. Had your portrait painted

48. Gone deep sea fishing

49. Seen the Sistine Chapel in person

50. Been to the top of the Eiffel Tower in Paris

51. Gone scuba diving or snorkeling

52. Kissed in the rain

53. Played in the mud

54. Gone to a drive-in theater

55. Been in a movie

56. Visited the Great Wall of China

57. Started a business

58. Taken a martial arts class

59. Visited Russia

60. Served at a soup kitchen

61. Sold Girl Scout Cookies

62. Gone whale watching

63. Got flowers for no reason

64. Donated blood, platelets or plasma

65. Gone sky diving

66. Visited a Nazi Concentration Camp

67. Bounced a check

68. Flown in a helicopter

69. Saved a favorite childhood toy

70. Visited the Lincoln Memorial

71. Eaten caviar

72. Pieced a quilt

73. Stood in Times Square

74. Toured the Everglades

75. Been fired from a job

76. Seen the Changing of the Guards in London

77. Broken a bone

78. Been on a speeding motorcycle

79. Seen the Grand Canyon in person

80. Published a book

81. Visited the Vatican

82. Bought a brand new car

83. Walked in Jerusalem

84. Had your picture in the newspaper

85. Read the entire Bible

86. Visited the White House

87. Killed and prepared an animal for eating

88. Had chickenpox

89. Saved someone’s life

90. Sat on a jury

91. Met someone famous

92. Joined a book club

93. Lost a loved one

94. Had a baby (almost)

95. Seen the Alamo in person

96. Swam in the Great Salt Lake

97. Been involved in a law suit

98. Owned a mobile phone

99. Been stung by a bee

100. Read an entire book in one day

n Friday morning all hell broke loose. The support team called me and were frantic that the website was broken. I immediately contacted the development team and asked them if they had made any unscheduled changes on Thursday night. They told me that they hadn’t made any changes since Monday afternoon. So this wasn’t a bug in the code it was a technical problem.

I went to our website to find out what was broken and everything I checked worked correctly. I contacted the support team again and asked them to explain exactly what was broken. This time they explained that only parts of the site were down so I asked them to send me a list of broken URLs. Since it takes me an hour to commute to the office I decided to hunker down and fix this problem from home.

Step 1 – Verify The Problem – This ensures that you understand what the problem is and that you are both talking about the same problem.

When I received the URLs they all worked. It is really hard to fix something if it is not broken. I contacted them again and it was still broken from the office. I tried connecting to the office via a VPN and when I checked the website as if I was located at the office it didn’t work. So I’ve been able to verify the problem but it seems to be limited to the office and a couple of customers which is a bit weird.

Step 2 – Occam’s Razor – Use Occam’s razor to come up with the simplest solutions and test each solution.

At this stage I decided there were three different things to test:
1. The computer at the office – Try using a different computer at the office – Same error.
2. The server – Try accessing the backup server from the office – No error.
3. The proxy at the office – Modified the host file at the office to switch the servers. – The error still only affect the live server.

Step 3 – Eliminate Options – Eliminate problems that aren’t backed up by the data and cross reference to figure out what the might cause the problem.

So at this stage I know:
1. From home I can access the backup site without errors.
2. From home I can access the live site without errors.
3. The office can access the backup site without any problems.
3. The office can access the live site but some URLs return blank pages or have their connections dropped.

The only difference in the data is that some URLs can’t be accessed from the office to the live server.

Step 4 – Isolate The Problem. – Try to find the boundary of what breaks and what doesn’t break.

At this stage I had no idea what the caused the error but I knew that if I could isolate exactly what was causing the problem then I might be able to fix it or work around the problem. I knew it wasn’t:
1. The users computer. (Elluminate because other computer at the same office had the same problem)
2. The proxy (A similar URL on the backup server worked)
3. The server (Everything worked from another location)

So I decided to try variations of the URL:
1. http://www.example.com/module/user.php?uid=123 – Was Broken – It returned a blank page at the office but the correct page from another location.
2. http://www.example.com/module/user.php – Worked – It displayed the same page at both locations.
3. http://www.example.com/module/user.php?id=123 – Worked – It displayed the same page at both locations.
4. http://www.example.com/index.php – Worked – It displayed the same page at both locations.
5. http://www.example.com/index.php?uid=789 – Was Broken – It returned a blank page at the office but the correct page from another location.

It took almost 30 minutes of trying different combinations to get obtain the 5 URLs above which enabled me to solve the problem. Based on the URLs above the problem it is possible to figure out that the problem is related to the query string (The part of the URL after the question mark). Specifically any query string that contained ‘uid’. I decided to try some more tests:

6. http://www.backup.com/index.php – Worked – It displayed the same page at both locations.
7. http://www.backup.com/index.php?uid=789 – Worked – It displayed the same page at both locations.

Combining the results of all these test allowed me to refine the problem to:

• There is a problem accessing the live server from the office using a URL that contains ‘uid’.

At this stage I was on a role and I felt like I was making progress. Can we isolate the problem in any other ways? Well what are the differences between the backup server and the live server?

• The OS is the same.
• The code is the same.
• The database is the same.
• The live server is located in Hong Kong.
• The backup server is located in China.

Maybe the problem is related to the location of the servers. Well that’s fairly easy to test:
8. http://www.google.com/?uid=123 (In America) – Was Broken – It returned a blank page at the office but the correct page from another location.
9. http://www.google.cn/?uid=123 (In China) – Worked – It displayed the same page at both locations.
10. http://www.adinobro.com/?uid=123 (My personal site in America) – Was Broken – It returned a blank page at the office but the correct page from another location.

I tested a number of other servers all over the world and basically any server outside of China was broken. Around this time some of the previous URLs started to work intermittently from the office.

The the problem can be defined as:

• There is a problem accessing the any server outside of china from the office using a URL that contains ‘uid’.

Here is a little bit of background information about the Internet in China:

The ‘Great Chinese Firewall’ is not actually a single firewall. Each ISP in China has different rules and filters traffic differently. While they all block traffic differently the end users experience is always the same – A blank web page is displayed or the browser reports that “The connection has been reset”. Also if the ISP has to much traffic then the excess traffic is often passed though without being filtered.

So after eliminating all the simple solutions the only solution left is that:

• The ISP that the office used was blocking and URLs to servers located outside of China (including Hong Kong) that contained ‘uid’ in the query string.

I did a final check and verified that the users that were having problems were using the same ISP that our office used. While I was on the phone with the customers the problem magically went away.

Why would the ISP block go out of it’s way to block ‘uid’? Well for a while some of us (Foreigner IT workers located in China) have suspected that Chinese ISP temporarily block access to sites and services not located in China to encourage companies to move their servers into a Chinese data center which cost significantly more and can be seized by the Chinese government. If they block the site in an obvious way then users will notice and get annoyed but if they block parts of the website then it looks like the website is broken. There is no way that companies can predict what will be broken next so if they want to be reliable then they have no choice but to move into a Chinese data center.

I was aware of these rumours but I thought Hong Kong would be treated as if it was part of China but not according to our ISP apparently.

So remember if you are trying to solve a problem:

1. Verify the problem by replicating it.
2. Occam’s Razor – Find the simplest explanations for the problem and test them.
3. Eliminate options that aren’t possible.
4. Isolate exactly what is causing the problem.

Once you have found the problem you can fix it.