This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!
by Sean M. Burke ISBN 0-596-00178-9 First Edition, published June 2002. (See the catalog page for this book.)
Search the text of Perl & LWP.
Table of Contents Copyright Page Foreword Preface Chapter 1: Introduction to Web Automation Chapter 2: Web Basics Chapter 3: The LWP Class Model Chapter 4: URLs Chapter 5: Forms Chapter 6: Simple HTML Processing with Regular Expressions Chapter 7: HTML Processing with Tokens Chapter 8: Tokenizing Walkthrough Chapter 9: HTML Processing with Trees Chapter 10: Modifying HTML with Trees Chapter 11: Cookies, Authentication, and Advanced Requests Chapter 12: Spiders Appendix A: LWP Modules Appendix B: HTTP Status Codes Appendix C: Common MIME Types Appendix D: Language Tags Appendix E: Common Content Encodings Appendix F: ASCII Table Appendix G: User's View of Object-Oriented Modules Index Colophon
Foreword I started playing around with the Web a long time ago—at least, it feels that way. The first versions of Mosaic had just showed up, Gopher and Wais were still hot technology, and I discovered an HTTP server program called Plexus. What was different was it was implemented in Perl. That made it easy to extend. CGI was not invented yet, so all we had were servlets (although we didn't call them that then). Over time, I moved from hacking on the server side to the client side but stayed with Perl as the programming language of choice. As a result, I got involved in LWP, the Perl web client library. A lot has happened to the web since then. These days there is almost no end to the information at our fingertips: news, stock quotes, weather, government info, shopping, discussion groups, product info, reviews, games, and other entertainment. And the good news is that LWP can help automate them all. This book tells you how you can write your own useful web client applications with LWP and its related HTML modules. Sean's done a great job of showing how this powerful library can be used to make tools that automate various tasks on the Web. If you are like me, you probably have many examples of web forms that you find yourself filling out over and over again. Why not write a simple LWP-based tool that does it all for you? Or a tool that does research for you by collecting data from many web pages without you having to spend a single mouse click? After reading this book, you should be well prepared for tasks such as these. This book's focus is to teach you how to write scripts against services that are set up to serve traditional web browsers. This means services exposed through HTML. Even in a world where people eventually have discovered that the Web can provide real program-to-program interfaces (the current "web services" craze), it is likely that HTML scraping will continue to be a valuable way to extract information from the Web. I strongly believe that Perl and LWP is one of the best tools to get that job done. Reading Perl and LWP is a good way get you started. It has been fun writing and maintaining the LWP codebase, and Sean's written a fine book about using it. Enjoy! —Gisle Aas Primary author and maintainer of LWP
Preface Perl soared to popularity as a language for creating and managing web content. Perl is equally adept at consuming information on the Web. Most web sites are created for people, but quite often you want to automate tasks that involve accessing a web site in a repetitive way. Such tasks could be as simple as saying "here's a list of URLs; I want to be emailed if any of them stop working," or they could involve more complex processing of any number of pages. This book is about using LWP (the Library for World Wide Web in Perl) and Perl to fetch and process web pages. For example, if you want to compare the prices of all O'Reilly books on Amazon.com and bn.com, you could look at each page yourself and keep track of the prices. Or you could write an LWP program to fetch the product pages, extract the prices, and generate a report. O'Reilly has a lot of books in print, and after reading this one, you'll be able to write and run the program much more quickly than you could visit every catalog page. Consider also a situation in which a particular page has links to several dozen files (images, music, and so on) that you want to download. You could download each individually, by monotonously selecting each link in your browser and choosing Save as..., or you could dash off a short LWP program that scans for URLs in that page and downloads each, unattended. Besides extracting data from web pages, you can also automate submitting data through web forms. Whether this is a matter of uploading 50 image files through your company's intranet interface, or searching the local library's online card catalog every week for any new books with "Navajo" in the title, it's worth the time and piece of mind to automate repetitive processes by writing LWP programs to submit data into forms and scan the resulting data.
0.1. Audience for This Book This book is aimed at someone who already knows Perl and HTML, but I don't assume you're an expert at either. I give quick refreshers on some of the quirkier aspects of HTML (e.g., forms), but in general, I assume you know what each of the HTML tags means. If you know basic regular expressions and are familiar with references and maybe even objects, you have all the Perl skills you need to use this book. If you're new to Perl, consider reading Learning Perl (O'Reilly) and maybe also The Perl Cookbook (O'Reilly). If your HTML is shaky, try the HTML Pocket Reference or HTML: The Definitive Guide (O'Reilly). If you don't feel comfortable using objects in Perl, reading Appendix G, "User's View of Object-Oriented Modules" in this book should be enough to bring you up to speed.
G.8. The Gory Details For sake of clarity of explanation, I had to oversimplify some of the facts about objects. Here's a few of the gorier details: ●
●
●
●
●
Every example I gave of a constructor was a class method. But object methods can be constructors, too, if the class was written to work that way: $new = $old->copy, $node_y = $node_x->new_subnode, or the like. I've given the impression that there's two kinds of methods: object methods and class methods. In fact, the same method can be both, because it's not the kind of method it is, but the kind of calls it's written to accept—calls that pass an object, or calls that pass a class name. The term "object value" isn't something you'll find used much anywhere else. It's just my shorthand for what would properly be called an "object reference" or "reference to a blessed item." In fact, people usually say "object" when they properly mean a reference to that object. I mentioned creating objects with constructors, but I didn't mention destroying them with destructor—a destructor is a kind of method that you call to tidy up the object once you're done with it, and want it to neatly go away (close connections, delete temporary files, free up memory, etc.). But because of the way Perl handles memory, most modules won't require the user to know about destructors. I said that class method syntax has to have the class name, as in $session = Net::FTP->new($host). Actually, you can instead use any expression that returns a class name: $ftp_class = 'Net::FTP'; $session = $ftp_class->new($host). Moreover, instead of the method name for object- or classmethod calls, you can use a scalar holding the method name: $foo->$method($host). But, in practice, these syntaxes are rarely useful.
And finally, to learn about objects from the perspective of writing your own classes, see the perltoot documentation, or Damian Conway's exhaustive and clear book Object Oriented Perl (Manning Publications, 1999).
Colophon Our look is the result of reader comments, our own experimentation, and feedback from distribution channels. Distinctive covers complement our distinctive approach to technical topics, breathing personality and life into potentially dry subjects. The animals on the cover of Perl and LWP are blesbok. Blesbok are African antelopes related to the hartebeest. These grazing animals, native to Africa's grasslands are extinct in the wild but preserved in farms and parks. Blesbok have slender, horselike bodies that are shorter than four feet at the shoulder. They are deep red, with white patches on their faces and rumps. A white blaze extends from between a blesbok's horns to the end of its nose, broken only by a brown band above the eyes. The blesbok's horns sweep back, up, and inward. Both male and female blesbok have horns, though the males' are thicker. Blesbok are diurnal, most active in the morning and evening. They sleep in the shade during the hottest part of the day, as they are very susceptible to the heat. They travel from place to place in long single-file lines, leaving distinct paths. Their life span is about 13 years. Linley Dolby was the production editor and copyeditor for Perl and LWP, and Sarah Sherman was the proofreader. Rachel Wheeler and Claire Cloutier provided quality control. Johnna VanHoose Dinse wrote the index. Emily Quill provided production support. Emma Colby designed the cover of this book, based on a series design by Edie Freedman. The cover image is a 19thcentury engraving from the Dover Pictorial Archive. Emma Colby produced the cover layout with QuarkXPress 4.1 using Adobe's ITC Garamond font. Melanie Wang designed the interior layout, based on a series design by David Futato. This book was converted to FrameMaker 5.5.6 with a format conversion tool created by Erik Ray, Jason McIntosh, Neil Walls, and Mike Sierra that uses Perl and XML technologies. The text font is Linotype Birka; the heading font is Adobe Myriad Condensed; and the code font is LucasFont's TheSans Mono Condensed. The illustrations that appear in the book were produced by Robert Romano and Jessamyn Read using Macromedia FreeHand 9 and Adobe Photoshop 6. This colophon was written by Linley Dolby.
Symbols & Numbers | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z
Index: B Babelfish, POST query example: 2.7. Example: Babelfish BBC headlines token example: 7.4.1. Example: BBC Headlines BBC News headline extraction, HTML::TreeBuilder: 9.4. Example: BBC News bookmark files, link extraction: 6.5. Example: Extracting Linksfrom a Bookmark File brittleness: 1.1.2. Brittleness browsers (see user agents) buttons radio buttons: 5.4.5. Radio Buttons reset: 5.4.8. Reset Buttons submit buttons: 5.4.6. Submit Buttons
Symbols & Numbers | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z
Symbols & Numbers | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z
Index: C can( ) method: 4.1.4. Components of a URL canonical, calling: 4.1.2. Output CGI (command gateway interface), formpairs.pl: 5.3. Automating Form Analysis Check tags code: 7.3.1. Checking Image Tags checkboxes: 5.4.4. Checkboxes children elements, siblings: 9.1. Introduction to Trees classes HTTP::Cookies::Netscape: 11.1.2. Loading Cookies from a File HTTP::Response: 3.1. The Basic Classes LWP class model: 3.1. The Basic Classes LWP::ConnCache: 3.4.1. Connection Parameters LWP::UserAgent: 3.1. The Basic Classes URI class: 4.1.1. Constructors cleanup, HTML::TreeBuilder: 9.2.4. Cleanup clone( ) method: 3.4. User Agents 4.1.1. Constructors code check tags: 7.3.1. Checking Image Tags detaching/reattaching nodes: 10.3. Detaching and Reattaching HTML::TreeBuilder: 9.2. HTML::TreeBuilder modifying attributes: 10.1. Changing Attributes tree example: 9.1. Introduction to Trees command-line utilities, formpairs.pl: 5.3. Automating Form Analysis comment tokens: 7.2.4. Comment Tokens comments access to, HTML::TreeBuilder: 10.4.2. Accessing Comments content, adding: 10.4.3. Attaching Content storage: 10.4.1. Retaining Comments comparing URLs: 4.1.3. Comparison components of regular expressions: 6.2.7. Develop from Components conn_cache( ) method: 3.4.1. Connection Parameters connection cache object: 3.4.1. Connection Parameters connection parameters, LWP::UserAgent class and: 3.4.1. Connection Parameters consider_response( ) function: 12.3.3. HEAD Response Processing 12.3.4. Redirects constructors: 4.1.1. Constructors HTML::TreeBuilder: 9.2.1. Constructors LWP::UserAgent class: 3.4. User Agents new( ): 4.1.1. Constructors new_from_lol( ): 10.5.2. New Nodes from Lists relative URLs and: 4.1.1. Constructors content( ) method: 3.5.2. Content content, adding to comments: 10.4.3. Attaching Content cookies: 11.1. Cookies
authentication and: 11.3.1. Comparing Cookies with Basic Authentication enabling: 11.1.1. Enabling Cookies HTTP::Cookies new method: 11.1.2. Loading Cookies from a File loading from file: 11.1.2. Loading Cookies from a File New York Times site example: 11.1.4. Cookies and the New York Times Site saving to file: 11.1.3. Saving Cookies to a File Set-Cookie line: 11.1. Cookies copyrights, distributions: 1.4.2. Copyright CPAN (Comprehensive Perl Archive Network): 1.3. Installing LWP CPAN shell, LWP installation: 1.3.1. Installing LWP from the CPAN Shell credentials( ) method: 3.4.5. Authentication current_age( ) method: 3.5.4. Expiration Times
Symbols & Numbers | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z
Symbols & Numbers | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z
Index: E elements HTML::Element: 10.5. Creating New Elements trees, attaching to other trees: 10.4. Attaching in Another Tree elements, trees: 9.1. Introduction to Trees children: 9.1. Introduction to Trees li elements: 9.1. Introduction to Trees tag comparison: 9.1. Introduction to Trees ul elements: 9.1. Introduction to Trees end-tag token: 7.1. HTML as Tokens 7.2.2. End-Tag Tokens end-tags, get_trimmed_text( ) method and: 7.5.4.2. End-tags env_proxy( ) method: 3.4.6. Proxies eq( ) method: 4.1.3. Comparison expressions (see regular expressions) extracted text, uses: 7.6. Using Extracted Text extracting data: 1.1.2. Brittleness regular expressions: 6.1. Automating Data Extraction extracting links, link-checking spider example: 12.3.5. Link Extraction
Symbols & Numbers | A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z
