The U.S. Census Bureau serves as the leading source of quality data about the nation's people and economy.
The U.S. Census Bureau's primary Web server consists of more than a million pages maintained by 60 disseminator groups of more than 200 disseminators. The Technology Applications Branch of the Systems Support Division needed to impose greater access control and accountability on a highly decentralized Web publishing model without losing any of the important features of the original system.
The original system allowed disseminators to publish directly to the U.S. Census Bureau's Internet Web server. Although this arrangement was fast and easy for the disseminators, it had several deficiencies:
The decision was made to implement a 3-tier solution:
The challenge was to ensure that the Web content on the production server was kept up-to-date. Provisions had to be made for disseminators to have the ability to quickly and easily push their changes from the staging system to the production system. Speed in publishing was a particular concern for Web publishers in the Economic Branch that publish the time-sensitive economic indicators, such as Retail Sales and New Home Sales.
As per the client's wishes, using freely-available open-source utilities such as Perl, rsync, and MySQL, a solution was developed, tested and implemented. Implementation was performed in an incremental manner as the groups of disseminators were trained by the Quotient on-site developer.
The solution consisted of constantly running server process on the staging system. The process continually scanned the Web directory hierarchy for either of two specially named service request files or token files. Once a disseminator places a token file in the directory that is to be synchronized, the following services are performed:
The system is flexible enough to run multiple requests in parallel.
With the new system:
With the efficiencies inherent in the rsync protocol and a program design that supports parallel processing, synchronization requests are satisfied very quickly, usually within 10 to 15 seconds.