Blog Of Fruits Chen

Bug: incorrect python sys.path

Thu 09 Feb 2012

So we happily upgraded a project to use some newer version of libs, and got it to use virtualenv. Until when I made changes, I found the new code doesn't make any difference. After some boring investigation I found the staging site is importing code from production site, so new code added to staging site is simply ignored. It's really important to avoid messing up with sys.path too much, like "sys.path.append('/path/to/live/site')". And when it is needed, make it dynamic path rather than hard coded. 

After fixing the hard coded path in the code, the issues remain. It proved to be /the-viertualenv/lib/python2.6/site-packages/easy-install.pth at last, some paths, somehow got added to this file. 

I never knew that easy-install.pth would also change sys.path. Something to read about the .pth files

Update: the built-in site lib can also be used to change your sys.path, example: 

import site
site
.addsitedir('/path/to/virtualenv/lib/python2.x/site-packages')

 

 

 

My Chinese website is dead, temprarily

Fri 24 Jun 2011

Just bought a tPad(a cheap Pad product by Teclast, a Chinese company) and wanna testing the browser. Typing my website. Oops, here goes the error message. Then I realized that my host provider put my site down because it doesn't have a "record". And it's been down for 2 weeks. And the Adsense income drops from $45 last month to $10 this month. 

So in China if you want to run a website, you need to get a record for your website. If the website belongs to a person, "they" need to know your name, id number, address, phone(work and mobile), etc. And after filling those info I found I need to go to a "check station" in person, bringing my ID, taking a photo and filling some more forms. I'm so lucky that there is a check station in my city! 

The website is built around 5 years ago. And I it's not been updated for nearly 2 years. I might considering giving it up if there weren't the adsense income, though not much. 

Start Reading Django Source Code

Sun 05 Sep 2010

I've been using Django for two years. And I always wanted to read the source of such a great framework. I think now is the time. Before start, I wrote a script to see that Django is made of. Here's the result:

.py 842 files(65.6786%) 91427 lines(87.4791%)
.js 27 files(2.1061%) 8722 lines(8.3454%)
.css 8 files(0.6240%) 2242 lines(2.1452%)
.html 83 files(6.4743%) 2122 lines(2.0304%)
2 files(0.1560%) 0 lines(0.0000%)
.vrt 1 files(0.0780%) 0 lines(0.0000%)
.png 7 files(0.5460%) 0 lines(0.0000%)
.mo 121 files(9.4384%) 0 lines(0.0000%)
.po 121 files(9.4384%) 0 lines(0.0000%)
.gif 39 files(3.0421%) 0 lines(0.0000%)
.json 3 files(0.2340%) 0 lines(0.0000%)
.csv 1 files(0.0780%) 0 lines(0.0000%)
.kml 2 files(0.1560%) 0 lines(0.0000%)
.shx 5 files(0.3900%) 0 lines(0.0000%)
.txt 2 files(0.1560%) 0 lines(0.0000%)
.prj 4 files(0.3120%) 0 lines(0.0000%)
.xml 4 files(0.3120%) 0 lines(0.0000%)
.dbf 5 files(0.3900%) 0 lines(0.0000%)
.shp 5 files(0.3900%) 0 lines(0.0000%)

 

The Python script to generate this table.

code_overview.py

 

import sys
import os.path
from collections import defaultdict
from pprint import pprint

ignore_dirs = ['nbproject']
total_files = 0
total_lines = 0
file_results = defaultdict(int)
file_ignores = []
line_results = defaultdict(int)
line_types = ['.py', '.js', '.css', '.html',]
final_results = []

def process_dir(data, dir, files):
    if os.path.basename(dir) in ignore_dirs:
        return
    print "processing %s" % dir
    for file in files:
        path = os.path.join(dir, file)
        if not os.path.isdir(path):
            process_file(path)

def process_file(path):
    global total_files, total_lines
    ext = os.path.splitext(path)[-1]
    file_results[ext] += 1
    total_files += 1
    if ext in line_types:
        line_count = len(open(path, 'r').readlines())
        line_results[ext] += line_count
        total_lines += line_count

def process_results():
    global total_files, total_lines
    for key, file_count in file_results.items():
        line_count = line_results[key]
        result = {
            'ext':key,
            'files':file_count,
            'lines':line_count,
            'file_percent': '%.4f%%' % (float(file_count)/float(total_files) * 100),
            'line_percent': '%.4f%%' % (float(line_count)/float(total_lines) * 100),
            }
        final_results.append(result)

def compare_lines(result1, result2):
    return result2['lines'] - result1['lines']

def sort_results():
    final_results.sort(compare_lines)

def print_results():
    sort_results()
    for result in final_results:
        s = '%(ext)s:\t %(files)s files(%(file_percent)s),\t %(lines)s lines(%(line_percent)s)' % result
        print s

def print_html_results():
    sort_results()
    table = []
    table.append('<table>')
    for result in final_results:
        s = '<tr><td>%(ext)s</td><td>%(files)s files(%(file_percent)s)</td><td>%(lines)s lines(%(line_percent)s)</td></tr>' % result
        table.append(s)
    table.append('</table>')
    print '\n'.join(table)

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print "Please provide the directory you want to process"
        sys.exit()
    os.path.walk(sys.argv[1], process_dir, None)
    process_results()
    if (len(sys.argv)>=3) and (sys.argv[2] == '-html'):
        print_html_results()
    else:
        print_results()

$ python code_overview.py ~/workspace/django/Django-1.2.1/django/ -html

 

 

 

Integrate Pygments, TinyMCE and Django

Wed 14 Jul 2010

Wrote a simple app, django_mce_pygments, to display source code with Pygments. To use the app:

  1. copy the plugin files into TinyMCE plugins directory. And include the plugin it in TinyMCE configuration
  2. add "django_pygements_mce" to settings and urls.
  3. include css file in the template where code is rendered. 

views.py as an exmple:

from django.http import HttpResponse
from django.template import RequestContext
from django.shortcuts import render_to_response

from pygments import highlight
from pygments.formatters import HtmlFormatter
from pygments.lexers import get_lexer_by_name

def pygments(request, template_name="pygments/pygments.html"):
if request.method == "POST":
lang = request.POST.get('lang', 'python')
code = request.POST.get('code', '')
code = highlight(code, get_lexer_by_name(lang),HtmlFormatter(cssclass="pygments"))
return HttpResponse(code)
else:
return render_to_response(template_name, {}
, context_instance=RequestContext(request))

 

A use case of Django Proxy Model with django-tagging

Mon 12 Jul 2010

I'm working on a project based on Pinax and our blog app is a bit customized. We've got three types of blogs and let's say they are type A, B and C. And we need to create three different tag clouds, for A, B and C respectively. django-tagging comes with Pinax and has an easy to use template tag "tag_cloud_for_model" to create tag clouds(see doc http://code.google.com/p/django-tagging/source/browse/trunk/docs/overview.txt).

{% tag_cloud_for_model [model] as [varname] %} 

The problem is it only accepts model as the input.

{% tag_cloud_for_model blog.Post as tags %} 

But what we need it to do is something like

{% tag_cloud_for_model blog.Post.filter(type=A) as tags %} 

{% tag_cloud_for_model blog.Post.filter(type=B) as tags %}


To avoid writing a custom "tag_cloud_for_model", Django Proxy models can help. Simply create a proxy model APost and overwrite its default queries.

class APostManager(models.Manager):
def get_query_set(self):
query = super(APostManager, self).get_query_set()
return query.filter(type=A)

class APost(Post):
objects = APostManager()
class Meta:
proxy = True

Then we can write

{% tag_cloud_for_model blog.APost as tags %} 

 

The life of a Satchmo Product

Wed 06 Jan 2010

Add a new product named "a tracking product"
# product.models.Product is saved.
On the category page, it's displayed.
# product.views.category_view
# product.models.Category.active_products()
Before it's displayed, Satchmo will try to find the best eligible discount for it and display it on category page.
# product.utils.find_best_auto_discount
It has its own page at /SHOP_BASE/product/a-tracking-product/
# product.views.get_product
Satchmo will see if it's a ProductVariation
# if 'ProductVariation' in product.models.Product.get_subtypes()
If it is, Satchmo will not show it
# product = product.productvariation.parent.product
Again, find eligible discount.
# product.utils.find_best_auto_discount(product)
Add context(options, variation details) to template
# product.models.Product.add_template_context
Find the appropriate template and display it.(detail_configurableproduct.html,...,fallback to product.html)
# product.views.find_product_template()

If there is a discount for this product, both original and new price will show
# product.templatetags.discount_price

The form to add this item to cart
# <form id="options" action="{% url satchmo_smart_add %}" method="post">
and some extensions
# {% plugin_point "product_add_buttons" %}
# {% plugin_point "product_form" %}
# {% plugin_point "product_footer" %}
for example
# satchmo_ext/productratings/templates/productratings/plugins/product_footer.html
# {% load satchmo_ratings %}
# {% product_ratings %}
# satchmo_ext/wishlist/templates/wishlist/plugins/product_add_buttons.html
# {% if user.is_authenticated %}{% load i18n %}
# <input type="submit" name="addwish" value="{% trans "Add to wishlist" %}" />
# {% endif %}

Most of the time, "add to cart" button is clicked.
# satchmo_store.shop.views.smart.smart_add
# satchmo_store.shop.signals.cart_add_view
# satchmo_store.shop.views.cart.add
But if "add to wish list" button is clicked,
# satchmo_ext.wishlist.listeners.wishlist_cart_add_listener
# satchmo_ext.wishlist.views.wishlist_add
Back to satchmo_store.shop.views.cart.add
# product, details = product_from_post(productslug, formdata)
# cart = satchmo_store.shop.models.Cart.objects.from_request(request, create=True)
# cart is possibly an order, if cartid == "order": cart = OrderCart(order)
# satchmo_store.shop.signals.satchmo_cart_details_query.send # but no one is listening.
# added_item = satchmo_store.shop.models.Cart.add_item(product, number_added=quantity, details=details)
The item is created, if the product has not been added previously.
# satchmo_store.shop.signals.satchmo_cart_add_verify.send
If configured, check if it's out of stock
# satchmo_store.shop.listeners.veto_out_of_stock
The CartItem does not store price, so if the price change, it will be reflected in the cart
# satchmo_store.shop.views.cart.display
Customer checkout
# payment.views.contact.contact_info_view
# HttpResponseRedirect(satchmo_utils.dynamic.lookup_url(paymentmodule, 'satchmo_checkout-step2'))
# payment module specific processing...
# ...
Create an order
# newOrder = Order(contact=contact)
Copy items from cart to order(CartItem to OrderItem)
# payment.utils.pay_ship_save
# payment.utils.update_orderitems
# satchmo_store.shop.signals.satchmo_post_copy_item_to_order.send(...)
Clean all items from the cart
# satchmo_store.shop.models.Cart.empty()
And about the price
# OrderItem.unit_price =>
# CartItem.unit_price =>
# CartItem.get_qty_price() and etc =>
# Product.get_qty_price()
# product.models.get_product_quantity_price, or product.models.get_product_quantity_adjustments() \ # or Product._get_fullPrice().
# price = get_product_quantity_price(self, qty) =>
# get_product_quantity_adjustments =>
# Price.adjustments() =>
# PriceAdjustmentCalc (the final_price)
 

Tracking the Code Flow of Twitter Post in Pinax

Thu 12 Nov 2009

In Pinax the user can post a tweet to their twitter account. As I need to write some code with the twitter APIs, I want to read the code of Pinax to find out how they did it. And I'll record it in this post.

Assume the username is 'hancock' and password is '123456'. Also, settings.SECRET_KEY = 'You can never guess this'.

First, the user submits his username and password to apps.account.views.other_services. Then the data submitted is handled by the form account.forms.TwitterForm. And before the data is saved, password will be encoded by the function microblogging.utils.get_twitter_password. Then account.models.update_other_services is called to save the data.

update_other_services(self.user,
twitter_user = self.cleaned_data['username'],
twitter_password = get_twitter_password(settings.SECRET_KEY, self.cleaned_data['password']),
)

def update_other_services(user, **kwargs):
"""
update the other service info for the given user using the given keyword args.
e.g. update_other_services(user, twitter_user=..., twitter_password=...)
"""
for key, value in kwargs.items():
info, created = OtherServiceInfo.objects.get_or_create(user=user, key=key)
info.value = value
info.save()


After this, two OtherServiceInfo instances will be saved.

info1.user = user
info1.key = 'twitter_username'
info1.value = 'hancock'

info2.user = user
info2.key = 'twitter_password'
info2.value = 'some string you won\'t be able to read' # encoded '123456'



Now that the user has a twitter account in the database, The user is now able to post a tweet to his twitter account at Pinax. The view to do this is microblogging.views.personal.

 

def personal(...)
twitter_account = twitter_account_for_user(request.user)
#......
twitter_account.PostUpdate(text)

def twitter_account_for_user(user):
profile = user.get_profile()
twitter_user = other_service(user, "twitter_user")
twitter_password = other_service(user, "twitter_password")
if twitter_user and twitter_password:
twitter_password = get_twitter_password(settings.SECRET_KEY, twitter_password, decode=True)
return twitter_account_raw(twitter_user, twitter_password)

def other_service(user, key, default_value=""):
try:
value = OtherServiceInfo.objects.get(user=user, key=key).value
except OtherServiceInfo.DoesNotExist:
value = default_value
return value

def twitter_account_raw(username, password):
return twitter.Api(username=username, password=password)



twitter_account_for_user is from microblogging.utils.py and it calls other_service from account.models.other_service to retrieve the user's username and password from twitter. (I think the two apps are coupled here. ) Then get_twitter_password is called again, but with decode set to True, to get the user's twitter decoded twitter password.

Then twitter.Api is called to get the user account and then the account is passed to the personal view. All that's left is to call twitter_account.PostUpdate(text).

Done and a bit hungry now.

 

Coclusion

I need to add a feature to highlight Python code in the post, rather than just make them italic. Yeah, the conclusion has nothing to do with the content with the post.

More

Say Thanks!

Wed 11 Nov 2009

Writing a blog in Django is really enjoyable. The blog have basic features such as tag, comment and archive. Though I wrote it myself, I cannot believe there are only 46 lines of code in the views.py, including many blank lines.So I really want to say thanks to the Django team and community. And also I want to say thank you to:

Kissy is a WYSIWYG editor released at  October 26th, 2009. Its file size is extremely small. And I integrated it in this blog very easily. Ihere is A blog application on Google App Engine. I read its source code before I started this blog.tagging, pagination and uni-form are all excellent 3rd party django apps, they are added to incorporate the correspondent features and to ease the development.Without these projects, this blog would be an impossible.