Fixing Generator Code

My last post was educational for me, but it turned out to be not quite correct.

There was a subtle issue that caused some confusion and an out-and-out bug. We’ll start with the bug first

return value of next

A friend pointed out to me that the usage of next was incorrect. Let’s look at the function we dissected last week:

def merge_lists_refactor(lists, labels):
    objects = list(map(next, lists))
    while any(x is not None for x in objects):
        i, obj = min(enumerate(objects), key=lambda x: (x[1]['time'], x[0]))
        yield labels[i], obj
        objects[i] = next(lists[i])

The behavior of the final line is where the bug crept in. The call to next is written assuming that next returns None when it’s finished generating values. This is not the case, of course it throws a StopIteration exception. At first I was a little confused at how I missed this. I had run the program over the dataset I was concerned with and it produced output.

That was the first mistake. Instead of generating a small, focused, meaningful test, I simply ran the code against two large log files. It generated output (lots of it – the log files were each about 20,000 lines long), which I deemed “correct”. The friend that pointed this out had tested with a clear data set which a small number of output lines which made it obvious that the program wasn’t running.

I was also confused as to how I didn’t see the exception bubbling up to the top and crashing my program. It turns out my test program was calling the generator as part of a for loop and the StopIteration exception was caught there.

for label, line in merge_lists_refactor([lines_one, lines_two],\
            [file_one, file_two]):

The communication issue

The other issue my reviewer had was one of poor communication on my part.

I not only didn’t provide a docstring for the function (mainly for brevity) I also did not show an example of calling it. He attempted to call the function with lists (which seems like a reasonable thing to do), but the function itself was designed to work with two generators, not two lists. The communication problem will be addressed below. Fixing the code to run on both lists and generators is left as an exercise for another day.

A generator to merge generators

Here’s the final version of the code, complete with example usage.

def merge_lists_refactor(lists, labels):
    objects = list(map(next, lists))
    while objects:
        i, obj = min(enumerate(objects), key=lambda x: (x[1]['time'], x[0]))
        yield labels[i], obj
        try:
            objects[i] = next(lists[i])
        except StopIteration:
            del objects[i]
            del labels[i]
            del lists[i]
list(merge_lists_refactor([iter(testa),iter(testb)], ['foo', 'bar']))

Note: there are many built-in solutions for this problem that are better for solving this. This is not intended as a “here’s how to do this” answer, but an educational journey.

Playing with Generators

I’ve noticed a common theme with many beginning-to-intermediate python developers is an uncertainty about how and when to use some language features. I have struggled with this as well and I’ve found the best way out of this is to force yourself to use the feature in small projects, even if it’s not the “best solution”. I recently worked through this exercise with generators and learned a lot!

In this post I’ll show you:

  • my simple solution using next with two generators
  • a cool solution using map, enumerate and lambda
  • a solution built-in to the standard library

Simple Generator Example

For this experiment, I had a simple problem for work. I wanted a script to merge two log files from different machines into a time-sorted file with each line labeled with the filename it came from. I know that it is possible to do this in bash much more quickly and easily, the point here is to play with generators, not bash.

I already had a generator which produced dictionaries from each line of the log file. For the purposes of this demo, the only key we care about in those dictionaries is ‘time’.

After playing a while, I came up with this solution(?). Unfortunately this solution is not quite correct as it exits as soon as one of the underlying generators is finished.

def merge_lists(list_one, label_one, list_two, label_two):
    """ Generate to return the earliest dated log entry from two lists """
    one = next(list_one)
    two = next(list_two)
    while one or two:
        if one['time'] < two['time']:
            yield label_one, one
            one = next(list_one)
        else:
            yield label_two, two
            two = next(list_two)

Ignoring the bug, the script works and demonstrates how a generator can pull from each of two different generators based on some criteria. The idea was to have an item from each file, compare the two, yield whichever was earlier (defaulting to file two on a tie) and then replace the element just yielded.

This is fairly straight-forward, but I was happy with it as a first attempt.

Better Generator Solution

I posted the above code as an example to a message board to which I belong and a friend there pointed out the bug and posted a much cooler solution.

def merge_lists_refactor(lists, labels):
    objects = list(map(next, lists))
    while any(x is not None for x in objects):
        i, obj = min(enumerate(objects), 
                     key=lambda x: (x[1]['time'], x[0]))
        yield labels[i], obj
        objects[i] = next(lists[i])

This solution not only doesn’t have the bug my original solution does, but it also generalizes to N files instead of just two.

This one took me a bit to figure out, so I’ll walk through it here.

map

Being well-versed in C and C++ syntax, the map function is one I have to think about for a minute. This usage:

    objects = list(map(next, lists))

runs next(list[i]) on each list in lists and returns a list of the objects returned. (try saying THAT three times quickly). It basically gets the first item from each list and creates a new list out of those items. The list() wrapping is required in Python 3 as map returns a generator instead of a list.

while any

The while loop is how he fixed the bug in my original code:

    while any(x is not None for x in objects):

This is a list comprehension used as a parameter to the any function. The list comprehension is:

    [x is not None for x in objects]

which produces a list of all items in the objects list which are not None.

The function any simply returns True if any of the items in the supplied list is True. So the loop will run until all lists are empty (or, put another way, until each item in objects is None).

min enumerate

The min/enumerate line took me a while to get through, I’ll admit.

        i, obj = min(enumerate(objects), 
                          key=lambda x: (x[1]['time'], x[0]))

There are really three parts to this line:

  • an enumerate call
  • a comparison function
  • a call to min()

The enumerate call produces a list of tuples in the form of (index, object) where object is at position ‘index’ in the list. This is pretty clear, even to me.

        enumerate(objects)

The min function is finding the minimum object in the given list. That list is the list of tuples generated above, so you need to give min a function to use for comparison. That’s what the lambda part is:

        i, obj = min(<generated list>, key=<comparison_function>)

Finally the lambda function is creating an anonymous function which pulls out the value from that data structure to use for comparison. In this case, each of the objects in the list looks like this:

(index, {'time': <datetime obj>, 'other_stuff':<otherstuff>})

So the syntax in the lambda:

        (x[1]['time'], x[0]))

translates

(index, {'time': <datetime obj>, 'other_stuff':<otherstuff>})

into

(<datetime obj>, index)

Since tuples compare in order, each item is compared by datetime object with ties being broken by the index in the list.

Reconstructing all of that line we get:

        i, obj = min(enumerate(objects), key=lambda x: (x[1]['time'], x[0]))

which creates a new list of tuples (timestamp, index) and then finds the minimum value in that list. The minimum tuple is then assigned to i, obj.

the rest

The final two lines of the loop are anti-climactic after all of that:

        yield labels[i], obj
        objects[i] = next(lists[i])

It just yields the label and obj and replaces the object we just used with a new object from the list selected.

Heapq

After all of this, another person on the forum chimed in that there is a function in the standard library that already does this:

list(heapq.merge(a,b, key=lambda....))

This is very likely to be faster for this operation, but it does fall short in that the names of the files would need to be prepended to the lines in a different manner.

It also suffers from the same problem I had with the bash solution: I wanted to play around with generators!

Thanks

Thanks for reading this far and thanks to Marc and Neal for their great ideas and encouragement!

Return of pylint

Until last fall I was working in python 2 (due to some limitations at work) and was very happy to have the Syntastic module in my Vim configuration to flag error each time I save a python file.  This was great, especially after writing in C/C++ for years where there is no official standard format and really poor tools to enforce coding standards.

Then, last fall when I started on Django, I made the decision to move to Python 3.  I quickly discovered that pylint is very version-dependent and running the python2.7 version of pylint against Python3 code was not going to work.

I wasn’t particularly familiar with virtualenv at the time, so I gave up and moved on with other things at the time.  I finally got back to fixing this and thus getting pylint and flake8 running again on my code.

Syntastic

I won’t cover the details of how to install Syntastic as it depends on how you manage your plugins in Vim and is well documented.  I will only point out here that Syntastic isn’t a checker by itself, it’s merely a plugin to run various checkers for you directly in Vim.  It run checkers for many languages, but I’m only using it for Python currently as the C code I use for work is so ugly that it will never pass.

Switching versions

The key to getting pylint to run against different versions of python is to not install pylint on a global level, but rather to install it in each virtualenv.  This seems obvious now that I’m more familiar with virtualenv, but I’ll admit it wasn’t at the time I first ran into the problem.

The other key to getting this to work is to only initiate Vim from inside the virtualenv.  This hampers my overall workflow a bit, as I tend to have gVim up and running for the long-term and just add files in new tabs as I go.  To get pylint to work properly, I’ll need to restart Vim when I switch python versions (at a minimum).  This shouldn’t be too much of a problem, however, as I’m doing less and less python2x coding these days.

Coding Style Thoughts

As I beat my head against horrible C code on a daily basis at work, I find myself appreciating more-and-more the idea of PEP-8 and having good tools for coding style enforcement.  While I frequently find some of the rules odd (two spaces here, but only one space there?) I really find it comforting to have a tool which runs, and runs quickly, to keep the code looking consistent.  Now if I could only get that kind of tool for C…….

 

Hello, Heroku!

This week I finally deployed the KidTasks app to Heroku.  While I was very happy with Heroku and how it behaved, I’ll have to admit that many of the shortcuts I’ve taken in developing this first app have come back to bite me during this process.  None of the issues are things I can blame Heroku for.  They were all self-inflicted.

Help!

To get started, I walked through the demo application instructions on Heroku and deployed the demo app they have.  I thought this demo was really well put together and explained things quite well.  Unfortunately, there were several requirements that I hadn’t yet met.  Fortunately, most of those were easily solved.

I had only been running the app locally and using the built-in Django server.  This is great for what it is, but not recommended for an actual deployment.

It’s Easy

The first several changes were fairly trivial.  They recommend using gunicorn; I’ll use gunicorn.  pip install; pip freeze; git add requirements.txt and that’s all set.

Similarly they recommend dj_database_url for reading the database settings out of the environment.  Also easily installed and the code modifications they showed worked without issue.

The White Album

The next suggested package was WhiteNoise which manages serving static content from Django.  I honestly had never thought of this before and had some learning to do.  The install was fine, but I ran into issues with the bootstrap fonts not being present during the collectstatic operation.  It took me a bit of digging but I figured out a solution for this – I put the bootstrap fonts in the static/bootstrap/fonts directory.  I’m not sure if this is the preferred solution, but it works for now and keeps all the files local.

At this point I attempted my first deploy.  Of course, it failed hard.

But the logs were kind enough to point out the problem:

CommandError: You must set settings.ALLOWED_HOSTS if DEBUG is False.

and, doing some searching on the web and the example app, the answer was easy, simply set the field to * in settings.py

# Allow all host headers
ALLOWED_HOSTS = ['*']

It was also at this point that it occurred to me that I had to make other changes to my settings.

Do You Want To Know A Secret?

Setting the debug flag to False was an obvious change in settings, but what to do about that pesky secret key?   Heroku (and the Two Scoops book) both recommend using an environment variable, but several sites on the web point out that environment variables are less secure than a file on the filesystem.  While this may be true in general (“cat /proc//environ” on a linux box will give you the environment variables), it doesn’t appear to be on Heroku’s dynos.

Another idea was to use a dynamic secret key for the app.  This can be done with code (from SO) that mirrors how Django initially creates these keys:

from django.utils.crypto import get_random_string

chars = 'abcdefghijklmnopqrstuvwxyz0123456789!@#$%^&*(-_=+)'
get_random_string(50, chars)

While this would work for this instance, there are some problems with it and it seems like using “best practices” as general rule would be a good idea, so I followed the directions and set it in the environment.

Mr. Post(gres)Man

Next came the big challenge.  I had been using sqlite3 for my app because, well, it’s easy.  Heroku doesn’t support sqlite3 as explained in the first answer in this SO post.

So, I needed to switch to a new database.  As postgres was the recommended solution and I didn’t have another preference, I went with that.

This conversion actually took a good portion of the entire process, again, largely through self-inflicted problems.

The basic setup for running postgres on Linux I gleaned from this site.

The dj_database_url and heroku’s default database config settings were fine.  There were just a couple of things I needed to learn first.  I needed to set up the postgres database with a full user, grant rights, etc.  Doing this locally, it took me a while to figure out all the the steps.  But, once I had done this, transferring that to Heroku was very straightforward.  You can to

heroku run bash

to get into the dyno, or for this case

heroku pg:psql

to get you directly into the postgres prompt on the production database.

Get Back

The final hurdle turned out to be with the database.  I had forgotten that you need to specify the app name when doing makemigrations and migrate.  I kept getting errors starting the app because it couldn’t find any of the required tables for the app.  It turns out that

heroku run python manage.py makemigrations
heroku run python manage.py migrate

acted like they succeeded but didn’t create the required tables.  After a bit of late-night head scratching, it finally dawned on me to add the name of the app

heroku run python manage.py makemigrations tasks
heroku run python manage.py migrate tasks

Once I got that solved the app was up and running.

Full Speed Ahead, Mr. Boatswain!

Mainly because I will likely find it handy in the future, I’m going to list some of the commands I found useful in the adventure.

  • heroku local web – test the configuration on your local system
  • heroku create kidtasks – create a new heroku app with a name
  • git push heroku master– this triggers a full deploy of the code
  • heroku config:set SECRET_KEY=<my super secret key>– sets environment variables on the app’s dyno
  • heroku run python manage.py [makemigrations tasks | etc] – basic django commands on the deployed app
  • heroku logs –tail – get the logs from the app

The End

While this process took more effort than I expected, I don’t think Heroku is to blame for any of it.  Now that I have a working example (and an idea of how postgres works), the next time will be much smoother.

Next app I start, I will definitely be using this template and see how much time it saves me.

Login, Please!

This week I was gearing up to try to deploy the KidsTask app. My first inclination was to try out Docker, mainly as it would be potentially useful knowledge for some work projects that are coming up.  That idea fell at the first hurdle.  I use an old laptop for my main development machine and Docker is only supported on 64 bit Linux.  While I did see ways to rebuild patched source to get Docker to run on a 32-bit system, that was more than I was interested in this week.

Next up: Heroku.  I’ve heard good things about this as a place to put very small apps and there are many resources for how to deploy a Django app to Heroku.  Putting the app on the external web, however, posed a new issue; KidsTasks didn’t have any form of authentication.  Anyone could go in and modify the data, which seems less-than-ideal in today’s world.

I (mistakenly) thought that adding user access to KidsTasks would be a good chunk of work.  After a little research and messing around, it turned out to be quite easy!  It turns out that the auth module shipped with Django was adequate for my needs and that configuring it was fairly straight forward.

Settings

You need to tell the auth module two pieces of information through the settings.  The first is where to go once a new user has logged in. This is specified by the name you supplied in the urlpatterns, not the url itself.  In my case this was just the “today” page which lists the daily tasks.  (NOTE: both of these setting are stand-alone variables inside the settings file, they are not part of any list or other structure)

LOGIN_REDIRECT_URL = 'today'

The next setting comes into play when you want to prevent users from accessing views without being logged in.  We’ll discuss how to set that up toward the end of this post, but for now, we’ll need to tell the auth module what the url is for the login page.

LOGIN_URL = 'login'

URLS

Next step is to add URLs to the urls.py file.  Most projects will have two or more urls.py files.  I chose the main project urls.py, rather than the one in my app, tasks, as those urls are prefixed by “tasks” and I wanted the login page to be reachable by just “login”.

We’re going to be using the built-in views in the auth module, so we need to import those first:

from django.contrib.auth import views as auth_views

Then we need to add the urls to the urlpatterns list.

urlpatterns = [
...
    url(r'^login/$', auth_views.login, name='login'),
    url(r'^logout/$', auth_views.logout, {'next_page': 'login'}, name='logout'),

Note the next_page parmaeter we’re passing to the logout view.  This is a nice little trick which forces a load of the logout url to log the user out and then redirect to the login page.

You must logout!

I almost forgot that we still need a way to get to the logout page.  I added a button to the navbar which is shown on all of the pages which points to logout.  Simple.

                  <li>
                  <a href="{% url 'logout' %}">Logout</a>
                  </li>

Login, please

The login template was by far the biggest section of work (and it wasn’t that large).  The template is very simple, just providing a form for the login information.  I’m still not happy with how it turned out, however.
I started by using the base.html like all of the other templates.  This, unfortunately put the navbar on the login page, which seemed odd.  So I ended up replicating some of the base.html code to get my css theme applied.  I’m still not happy with this solution, however, as I suspect there should be a better way not repeat code.  I might explore some of these options at a later point.

Another issue with the login template is that it wants to live in the ‘registration’ app by default.  Rather than figuring out how to change the default, I simply added a registration directory under the tasks app’s templates directory.  (I started by creating a top-level registration app which simply had the template in it.   I liked this even less.).  This is another ‘technical-debt’ issue that I hope to clean up in the future.

The template itself is seen below.  Nothing too shocking; basic html wrapper, bootstrap initialization, and a form.

<html>
<div>
   <head>
   {% load bootstrap3 %}
   {% bootstrap_css %}
   {% bootstrap_javascript %}
   {% bootstrap_messages %}

   <title>KidTasks</title>
   </head>
   <body>

   <div class="page-header">
      <h1>{% block title %}KidTasks Login{% endblock %}</h1>
   </div>

   <div>
      <form method="post">
         {% csrf_token %}
         {% bootstrap_form form %}
         {% buttons %}
         <button type="submit" name="save" value=True class="btn btn-success">Login</button>
         {% endbuttons %}
      </form>
   </div>
   </body>
</div>
</html>

Login Required!

Once you’ve done all of the above, you’ll be able to log in to your app.  The app will still allow you to access all of the pages without logging in.  It turns out there is a remarkably clever and simple solution for this: the @login_required decorator.

In your views file, you’ll need to import

from django.contrib.auth.decorators import login_required

and then add this decorator to each view function you want to be restricted.

@login_required

Works like a charm!

While I left a couple of loose ends I’ll eventually have to clean up, the task of adding login restrictions to KidsTasks was much easier than I expected.

The Case of the Forgotten Space

I spent some time this week doing a little clean up on KidTasks, mainly converting the forms to use bootstrap, sharing some repeated code, and adding a little functionality.

All of this is prep for my first round of “user testing”.  Since this is essentially an app I’ve written for my family to use, the testing will be in-house, to say the least.  I plan on hosting it from my laptop for a bit (mainly to keep things simple) but eventually I hope to move it to Heroku or something similar.  That’s a topic for a different post.

Most of the changes this week (visible here) are mundane changes, but I did add a set of features which posed an interesting problem for me.

The Feature

After doing a bunch of debugging and testing on my database, it (finally) occurred to me that having a way to change a kid’s name and delete a kid could be quite handy functionality.  This mainly came about after I noticed that I had shared template code between kid_edit and repeating_task_edit templates.  There was a repeating_task_update.html, but no kid_update!  That was the missing functionality.

Now, to be fair, code to change or delete kids won’t likely get run very often once the system is set up, but it seemed like an obvious thing to have, and, a good way to get my mental gears running on Django again.

The Problem

Well, I coded it up, added a link in the navbar to a new page listing all the kids with links for each to the kid_update template.  So far, so good.

I worked up the kid_update template (not hard at all) and tested it out.  I tried to delete a kid and, presto-chango, the kid was removed.  It was late and I was wiped out and I almost gave up work for the evening.  Just one more test…..

Let’s change the name of the kid from “Jim” to “Jim Anderson” (note: the names of the kids have been changed to protect the innocent.)  Of course, you have already guessed that this test failed.

Well, I was still tired and did put it to rest of the evening as I knew I wouldn’t solve it that night.

Some Wrong Solutions

I had a few minutes the next morning before the kids got up and started looking into the issue.  The problem was that Django could not find a matching URL after I changed the name. At the time, the idea came to me that Django was unhappy because the name was used as a primary key in the database and it couldn’t find the record.  As I said, I only had a few minutes and that was about as far as I got.   I spent many of my free cycles at work wondering how much effort it was going to cost me to fix this.

Finally that evening I got some time to get back to the problem (it had been annoying me all day!).  I quickly remembered what the Django experts among you already know: Django creates an ID field for you (unless you do something special) as the primary key.

That wasn’t the problem.  Well, it’s complaining about not finding the URL, but I can see the url in the set of URLs it displays in debug mode.  What was going on here?

The url line I had added for this page was

    url(r'^kid/update/(?P<name>[a-zA-Z]+)$',
        views.update_kid, name='kid_update'),

Aha! That regex I used to match the kid’s name didn’t include spaces!  That was the real problem.  I added a space to the regex:

    url(r'^kid/update/(?P<name>[a-zA-Z ]+)$',
        views.update_kid, name='kid_update'),

It worked!  Problem solved.

Except something was nagging at me about this solution.  It wouldn’t work with Unicode names, it wouldn’t work with any punctuation, hmm.  This was going to be harder than I thought.  There had to be a better answer.

The Solution

Finally, the light dawned and I saw that I was using the wrong attribute of the model for the URL lookup. Use the ID, of course!

    url(r'^kid/update/(?P<kid_id>[0-9]+)$',
        views.update_kid, name='kid_update'),

After that all I had to do was change the URL, the referring templates, and the view that handles it, and all was back to working, without the restrictions of my previous answer.

I’m sure there are more mistakes like this lurking in this project, but hopefully I’ll iron them out as we move through the process and learn.

Deep dive into manage.py

I don’t know about you, but when I started with Django, I wondered about the “manage” script and what it did.  But there were too many other things to learn, so I accepted that it worked and moved on, using the recipes I found in examples and documentation.

My curiosity has gotten the better of me, so now I’m going to take a break and examine what’s going on under the hood.

Naively, I had assumed that the manage script was some large, static beast with 1000’s of lines of code.  (Too much C and make in my background, I guess).  Of course, this is python, so manage.py is small, reflective, and quite dynamic.

On a brand new project, manage.py weighs in at a whopping 22 lines of code.  Now, that’s not really a fair accounting as all that code is doing is importing a method from a module:

from django.core.management import execute_from_command_line

and then running it:

execute_from_command_line(sys.argv)

Th execute_from_command_line function is located in the __init__.py package in python/site-packages in your virtualenv   (you are using virtualenv, right?).

This app does several bookkeeping things (managing command line args, deciding if it needs to do “help”, etc) and then calls the get_commands function.  This is a very elegant function (the comment block explaining it is much longer than the code) that creates a { command_name : app_name } dictionary for all of the commands it can find.  It starts by adding all of the django core commands by searching the management/commands directory in the core django directory.  It then searches the management/commands subdirectory of each installed app.

It took a little digging, but I managed to figure out a bit more detail here which I found interesting.  Django has a list called django.apps.apps.  These are the apps listed in the INSTALLED_APPS list in your settings.py file.  An entry is made for each app loaded with some details about the app.  For our purposes here, the data that we care about is the path and the name.

Jumping back to the get_commands function and the __init__.py file in the django.core.management module, you can see at the top of the file that it imports this list of apps:

from django.apps import apps.

This allows the get_commands function to walk the list of apps, getting the name of each app and the path in which it’s stored.  It appends the management/commands subdirectory to the path and adds each python file there as a command to the { command_name : app_name }, listed above.  (Note that I’m skipping some details here).  That’s how it figures it out.

So, to complete the loop, we can see that the apps listed in settings.py

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
]

show up in the lib/python3.4/site-packages/ directory structure:

django/
├── core
│   ├── management
│   │   ├── commands
│   │   │   ├── check.py
│   │   │   ├── compilemessages.py
│   │   │   ├── createcachetable.py
│   │   │   └── [several commands removed]
│   │   │   ├── runserver.py
│   │   │   └── [several commands removed]
├── contrib
│   ├── auth
│   │   ├── management
│   │   │   ├── commands
│   │   │   │   ├── changepassword.py
│   │   │   │   ├── createsuperuser.py
│   ├── contenttypes
│   │   ├── management
│   │   │   ├── commands
│   │   │   │   └── remove_stale_contenttypes.py
│   ├── sessions
│   │   ├── management
│   │   │   ├── commands
│   │   │   │   ├── clearsessions.py
│   ├── staticfiles
│   │   ├── management
│   │   │   ├── commands
│   │   │   │   ├── collectstatic.py
│   │   │   │   ├── findstatic.py
│   │   │   │   └── runserver.py

and are represented in the output of ./manage.py –help

Available subcommands:

[auth]
    changepassword
    createsuperuser

[contenttypes]
    remove_stale_contenttypes

[django]
    check
    compilemessages
    createcachetable
    [several commands removed but NOT runserver]

[sessions]
    clearsessions

[staticfiles]
    collectstatic
    findstatic
    runserver

So, when you add django-extensions to your INSTALLED_APPS list, that is how the new commands show up!

Those of you that are really detail-oriented might have noticed that the runserver command is listed in both the django.core app as well as in the staticfiles app.  It is only listed once in the command output.  It turns out that the way the dictionary of commands is constructed, it uses the command name for the key and the app name for the value.  This means that the last one found will be the command listed.  This works out well in this case as the staticfiles version of the command add functionality you very likely want (mainly serving static files) on top of the core runserver command.  In classic DRY style, the staticfiles version of the command makes use of the core version.  It also has the nice effect of providing a runserver command which does NOT serve static files if you do not have the django.contrib.staticfiles app installed.  Sweet!

Building your own extensions

If you follow all of that, the documentation that describes adding your own management commands for your app should seem fairly obvious.  Basically you just create a management/commands directory in your app and populate it with the commands you wish to add.  Note that there are a few other requirements (there needs to be an __init__.py file, the commands must all be subclasses of django.core.management.base.BaseCommand, etc), but the basic mechanism is exactly the same.

Django, Bootstrap, and you!

Last post I talked about how to get bootstrap integrated into your Django app.  This week I want to dive a little deeper into how to use bootstrap themes and customize them easily.

While the default bootstrap theme that Django-bootstrap uses by default is much nicer than I could design, there are many, many options on the web for different bootstrap themes.  In this post we’re going to pull in a different theme, and then figure out how to modify with minimal effort.

Loading Different Themes

There are many sites containing free (and paid!) bootstrap themes which you can try out, download and modify until you get the look you want.  A quick search this evening produced several hits.  I’ll be using bootswatch in this example, but other sites should work in similar manners.
The mechanics of this are actually quite simple.  There’s a dictionary that django-bootstrap3 uses from your settings.py file called, wait for it….
BOOTSTRAP3!
The Django-bootstrap documentation lists all of the various things you can set here, but the one we’re concerned with here is the theme_url setting.
So, say you went to bootswatch and thought you’d give the superhero theme a try.  The “Download” button on the site lists a bunch of options, only the first two are of interest to us today; bootstrap.min.css and bootstrap.css.

The two files should be the same with the exception that the .min. file has all of the whitespace removed from the css portion of it. (Usually the header comments still have whitespace.)   This is done to minimize the amount of data that needs to be served on high-volume sites.   If you’re trying to read or edit a theme file, you’ll want the regular version.    There are several websites that will help you minimize your css when you’re ready to deploy.

Try downloading both of the css files and looking at them.

When you download these files, you’ll want to store them in the static files directory of your Django app.  For my app the app directory is tasks, so the full path of where the min file is stored is tasks/static/bootstrap/css/bootstrap.min.css.  (The non-min version is stored in the same directory).

Once you’ve downloaded them, you tell the Django-Bootstrap3 module to use that with the following setting:

BOOTSTRAP3 = { 'theme_url': '/static/bootstrap/css/bootstrap.min.css', }

Don’t be a Jerk

While you’re playing around with different themes, you can simply use the URL they have for the download and put it into your settings like this:

BOOTSTRAP3 = { 'theme_url':
              'https://maxcdn.bootstrapcdn.com/bootswatch/3.3.7/superhero/bootstrap.min.css', }

But don’t be a jerk.  If you’re actually going to use the theme, download it and serve it statically from your site.  The people providing these themes shouldn’t have to pay to serve content for your web app.  I think it’s fine to use it for trying out a bunch of themes, though.

Customizing your Theme

When you decide to customize the theme you’ve downloaded, I’ll pass on advice I read somewhere doing research for this post.  Rather than just editing the downloaded theme’s css file, you can take advantage of the “cascading” part of CSS and create a file which just has your changes in it.  This keeps your changes separate from the theme which makes it much easier to change themes or roll to a new version of the same theme if and when it becomes available.

I’ve found two ways to do this, and I’m frankly not sure which is “better” or more pythonic.

Method One

The method I figured out first was to use two different ways to include a CSS file into your app.  The BOOTSTRAP dictionary, shown above, gives you the first.  In this method you use that to point to the unmodified theme file.  Then, in the base.html template (which is now included in all other templates) you manually include the link to your changes file.  This makes the bootstrap section of my base.html file look like:

   {% load bootstrap3 %}
   {% bootstrap_css %}
   {% bootstrap_javascript %}
   {% bootstrap_messages %}
   <link rel="stylesheet" type="text/css" href="{% static 'bootstrap/css/mybootstrap.css' %}" />

While this spreads the information about which css files you’re using around the system a bit more (which I dislike), it keeps your modifications completely independent of the base theme file (which I like).

Method Two

This method actually came to me while I was writing up this post.  CSS allows your to include other CSS files in them.  It turns out, if the included file is in the same directory, the syntax is pretty straightforward.   Simply add this line to the top of your css file (it must be the first line, I believe).

@import url("superhero.min.css");

You’ll then need to have your changes file in the BOOTSTRAP dictionary in settings.py:

BOOTSTRAP3 = { 'theme_url': '/static/bootstrap/css/mychanges.css', }

and, if superhero.min.css is in the same directory as your mychanges.css, it will just work!

Looking at it more, it feels like this solution is a little “cleaner”, but I’m interested to hear if you have an opinion.

KidTasks Restart – Bootstrap!

Snowboard season is just about finished, so it’s time to get back to Python, Django, and blogging!

During the snowboard season I spent a little time learning CSS and then figuring out how to apply it to a Django project.  I went through a couple of iterations on this, starting with adding CSS directly to the project (as a static file) and then using Bootstrap directly (as shown in this nice blog post)  Finally, I found the very nice Django-Bootstrap3 component which simplifies the process nicely!

While the documentation is quite good, I did miss a simple step which threw me for a bit: you must add ‘bootstrap3’ to you list of INSTALLED_APPS in your settings.py file.  I’ll admit that it says it right there on the installation page listed above, but I somehow missed it.

Once I got past that self-inflicted hurdle, things worked really smoothly! Adding the following:

   {% load bootstrap3 %}
   {% bootstrap_css %}
   {% bootstrap_javascript %}
   {% bootstrap_messages %}

to the <head> section of a template allows your to use the bootstrap CSS classes to really produce nice output quite easily!

While playing with this and the various classes in Bootstrap, I was struck with how sloppy the templates I had created were and took a few minor steps (and applied the DRY principle) to clean it up.  The biggest part of the cleanup was to create a base.html and have all of my templates extend the base template via the:

    {% extends 'tasks/base.html' %}

line at the top.  This allowed me to use the navbar class from bootstrap to have a nice set of navigation links on the top of every page (and thus get rid of the copy-paste code I had used to add these links previously!)

The code for the navbar looks like this, using the Django {% url } call to get navigation links inside the app.

   <nav class="navbar navbar-default navbar-fixed-top">
      <div class="container-fluid">
         <div class="navbar-header">
            <a class="navbar-brand" href="#">KidTasks</a>
         </div>

         <div class="collapse navbar-collapse">
            <ul class="nav navbar-nav">
               <li>
               <a href="{% url 'task_new' %}?from={{ request.path|urlencode }}">Add New Task</a>
               </li>
               <li>
               <a href="{% url 'rep_task_new' %}?from={{ request.path|urlencode }}">Add New Repeating Task</a>
               </li>
               <li>
               <a href="{% url 'kid_new' %}?from={{ request.path|urlencode }}">Add New Kid</a>
               </li>
               <li>
               <a href="{% url 'today' %}">View Today's Tasks</a>
               </li>
            </ul>
         </div><!-- /.navbar-collapse -->
      </div><!-- /.container-fluid -->
   </nav>

That’s all there is to it!  Those of you curious to see what this looks like
can check out the code at github:

git@github.com:jima80525/KidTasks.git
git checkout blog/06-Bootstrap

As I said at the top, I’m hoping to get back into the swing of studying and blogging again.  Look for more next week!

Python, Snowboarding, and time

It’s mid-snowboard season here in North America, so time for learning, researching and writing is scarce.  I wanted to post to keep the blog alive and to give a shout to a friend who’s starting up a business selling some cool python swag.

I’m not in marketing so I won’t even pretend to give a pitch, other than to say I bought a mug (from coffee import *) and love it!

If you’re looking for something unique to give to a python geek (like yourself!) check it out at NerdLettering

OK.  I promise this blog won’t turn into a sales platform AND I promise to be back in the spring with more learning in python and django.  But for now, let it SNOW!!!!