GitHub - appscluster/python-readability: Fast python port of arc90's readability tool, updated to match latest readability.js!

appscluster / python-readability Public

forked from buriy/python-readability

Notifications You must be signed in to change notification settings
Fork 0
Star 0

Fast python port of arc90's readability tool, updated to match latest readability.js!

github.com/appscluster/python-readability

0 stars 461 forks Branches Tags Activity

Star

Notifications

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 82 Commits
readability		readability
tests		tests
.gitignore		.gitignore
Makefile		Makefile
README		README
setup.py		setup.py

Repository files navigation

Python-Readability-lxml Version 0.3.0.7
=======================================

Forked from Yuri Baburov <burchik@gmail.com>

This code is under the Apache License 2.0.  http://www.apache.org/licenses/LICENSE-2.0

This is a python port of a ruby port of arc90's readability project

http://lab.arc90.com/experiments/readability/

In few words,
Given a html document, it pulls out the main body text and cleans it up.
It also can clean up title based on latest readability.js code.

Based on:
 - Latest readability.js ( https://github.com/MHordecki/readability-redux/blob/master/readability/readability.js )
 - Ruby port by starrhorne and iterationlabs
 - Python port by gfxmonk ( https://github.com/gfxmonk/python-readability , based on BeautifulSoup )
 - Decruft effort to move to lxml ( http://www.minvolai.com/blog/decruft-arc90s-readability-in-python/ )
 - "BR to P" fix from readability.js which improves quality for smaller texts.
 - Github users contributions.


Installation::

Dev Version:

    wget https://github.com/appscluster/python-readability/archive/master.zip

    7z x master.zip ( if you don't have 7z : sudo apt-get install p7zip-full )

    cd python-readability-master/

    python setup.py install

Base Version:

    easy_install readability-lxml
    or
    pip install readability-lxml


Usage::

    from readability.readability import Document
    import urllib
    html = urllib.urlopen(url).read()
    readable_article = Document(html).summary()
    readable_title = Document(html).short_title()


Command-line usage::
    
    Default: 

    python -m readability.readability -u http://pypi.python.org/pypi/readability-lxml

    Add User-Agent: 

    python -m readability.readability -u http://www.bbc.co.uk/newsbeat/30520643 -a "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36" 


Using positive/negative keywords example::

    python -m readability.readability -p intro -n newsindex,homepage-box,news-section -u http://python.org


Document() kwarg options:

 - attributes:
 - debug: output debug messages
 - min_text_length:
 - retry_length:
 - url: will allow adjusting links to be absolute
 - positive_keywords: the list of positive search patterns in classes and ids, for example: ["news-item", "block"]
 - negative_keywords: the list of negative search patterns in classes and ids, for example: ["mysidebar", "related", "ads"]


Updates

 - 0.3.0.7 Encoding updated based on buriy's update
 - 0.3.0.6 Added option for User-Agent inclusion