I really should have posted about this earlier but life has been a little bit hectic over the last couple of days. On the 13th of March 2009 Zander James Brown was born. Enjoy the picutres:
Easter Zander

Dad & Zander

Mum, Dad & Zander

|
I really should have posted about this earlier but life has been a little bit hectic over the last couple of days. On the 13th of March 2009 Zander James Brown was born. Enjoy the picutres: Easter Zander
Dad & Zander
Mum, Dad & Zander
This is a meme that is going around and it looks interesting. A lot of it is very American but that’s life. I first read it at Empty Thoughts. 1. Started your own blog 2. Slept under the stars 3. Played in a band 4. Visited Hawaii (Visited Vanuatu & Malaysia which are closer to Australia but similar) 5. Watched a meteor shower (Tried to 3 times but failed) 6. Given more than you can afford to charity 7. Been to Disneyland 8. Climbed a mountain 9. Held a praying mantis 10. Sang a solo 11. Bungee jumped 12. Visited Paris 13. Watched a lightning storm 14. Taught yourself an art from scratch 15. Adopted a child 16. Had food poisoning 17. Walked to the top of the Statue of Liberty 18. Grown your own vegetables 19. Seen the Mona Lisa in France 20. Slept on an overnight train 21. Had a pillow fight 22. Hitch hiked 23. Taken a sick day when you’re not ill 24. Built a snow fort 25. Held a lamb 26. Gone skinny dipping 27. Run a Marathon 28. Ridden in a gondola in Venice 29. Seen a total eclipse 30. Watched a sunrise or sunset 31. Hit a home run 32. Been on a cruise 33. Seen Niagara Falls in person 34. Visited the birthplace of your ancestors 35. Seen an Amish community 36. Taught yourself a new language (Mandarin) 37. Had enough money to be truly satisfied 38. Seen the Leaning Tower of Pisa in person 39. Gone rock climbing 40. Seen Michelangelo’s David 41. Sung karaoke 42. Seen Old Faithful geyser erupt 43. Bought a stranger a meal at a restaurant 44. Visited Africa 45. Walked on a beach by moonlight 46. Been transported in an ambulance 47. Had your portrait painted 48. Gone deep sea fishing 49. Seen the Sistine Chapel in person 50. Been to the top of the Eiffel Tower in Paris 51. Gone scuba diving or snorkeling 52. Kissed in the rain 53. Played in the mud 54. Gone to a drive-in theater 55. Been in a movie 56. Visited the Great Wall of China 57. Started a business 58. Taken a martial arts class 59. Visited Russia 60. Served at a soup kitchen 61. Sold Girl Scout Cookies 62. Gone whale watching 63. Got flowers for no reason 64. Donated blood, platelets or plasma 65. Gone sky diving 66. Visited a Nazi Concentration Camp 67. Bounced a check 68. Flown in a helicopter 69. Saved a favorite childhood toy 70. Visited the Lincoln Memorial 71. Eaten caviar 72. Pieced a quilt 73. Stood in Times Square 74. Toured the Everglades 75. Been fired from a job 76. Seen the Changing of the Guards in London 77. Broken a bone 78. Been on a speeding motorcycle 79. Seen the Grand Canyon in person 80. Published a book 81. Visited the Vatican 82. Bought a brand new car 83. Walked in Jerusalem 84. Had your picture in the newspaper 85. Read the entire Bible 86. Visited the White House 87. Killed and prepared an animal for eating 88. Had chickenpox 89. Saved someone’s life 90. Sat on a jury 91. Met someone famous 92. Joined a book club 93. Lost a loved one 94. Had a baby (almost) 95. Seen the Alamo in person 96. Swam in the Great Salt Lake 97. Been involved in a law suit 98. Owned a mobile phone 99. Been stung by a bee 100. Read an entire book in one day n Friday morning all hell broke loose. The support team called me and were frantic that the website was broken. I immediately contacted the development team and asked them if they had made any unscheduled changes on Thursday night. They told me that they hadn’t made any changes since Monday afternoon. So this wasn’t a bug in the code it was a technical problem. I went to our website to find out what was broken and everything I checked worked correctly. I contacted the support team again and asked them to explain exactly what was broken. This time they explained that only parts of the site were down so I asked them to send me a list of broken URLs. Since it takes me an hour to commute to the office I decided to hunker down and fix this problem from home. Step 1 – Verify The Problem – This ensures that you understand what the problem is and that you are both talking about the same problem. When I received the URLs they all worked. It is really hard to fix something if it is not broken. I contacted them again and it was still broken from the office. I tried connecting to the office via a VPN and when I checked the website as if I was located at the office it didn’t work. So I’ve been able to verify the problem but it seems to be limited to the office and a couple of customers which is a bit weird. Step 2 – Occam’s Razor – Use Occam’s razor to come up with the simplest solutions and test each solution. At this stage I decided there were three different things to test: Step 3 – Eliminate Options – Eliminate problems that aren’t backed up by the data and cross reference to figure out what the might cause the problem. So at this stage I know: The only difference in the data is that some URLs can’t be accessed from the office to the live server. Step 4 – Isolate The Problem. – Try to find the boundary of what breaks and what doesn’t break. At this stage I had no idea what the caused the error but I knew that if I could isolate exactly what was causing the problem then I might be able to fix it or work around the problem. I knew it wasn’t: So I decided to try variations of the URL: It took almost 30 minutes of trying different combinations to get obtain the 5 URLs above which enabled me to solve the problem. Based on the URLs above the problem it is possible to figure out that the problem is related to the query string (The part of the URL after the question mark). Specifically any query string that contained ‘uid’. I decided to try some more tests: 6. http://www.backup.com/index.php – Worked – It displayed the same page at both locations. Combining the results of all these test allowed me to refine the problem to:
At this stage I was on a role and I felt like I was making progress. Can we isolate the problem in any other ways? Well what are the differences between the backup server and the live server?
Maybe the problem is related to the location of the servers. Well that’s fairly easy to test: I tested a number of other servers all over the world and basically any server outside of China was broken. Around this time some of the previous URLs started to work intermittently from the office. The the problem can be defined as:
Here is a little bit of background information about the Internet in China: The ‘Great Chinese Firewall’ is not actually a single firewall. Each ISP in China has different rules and filters traffic differently. While they all block traffic differently the end users experience is always the same – A blank web page is displayed or the browser reports that “The connection has been reset”. Also if the ISP has to much traffic then the excess traffic is often passed though without being filtered. So after eliminating all the simple solutions the only solution left is that:
I did a final check and verified that the users that were having problems were using the same ISP that our office used. While I was on the phone with the customers the problem magically went away. Why would the ISP block go out of it’s way to block ‘uid’? Well for a while some of us (Foreigner IT workers located in China) have suspected that Chinese ISP temporarily block access to sites and services not located in China to encourage companies to move their servers into a Chinese data center which cost significantly more and can be seized by the Chinese government. If they block the site in an obvious way then users will notice and get annoyed but if they block parts of the website then it looks like the website is broken. There is no way that companies can predict what will be broken next so if they want to be reliable then they have no choice but to move into a Chinese data center. I was aware of these rumours but I thought Hong Kong would be treated as if it was part of China but not according to our ISP apparently. So remember if you are trying to solve a problem:
Once you have found the problem you can fix it. |