Late, half-baked crap

Recently, I had a reminder that overall, a lot of things just don’t get done.

It’s rarely a conspicuous lack of a thing getting done that reminds me — instead, it’s the surprised, positive reaction from people when things do get done. We simply couldn’t muster such shock and joy if most things just… got finished.

It’s easy to assume that other people get stuff done. Certainly people finish stuff at work, do taxes, and fill out medical forms. But outside that — keeping up with friends and colleagues, participating in clubs and sports, or working on side projects — it’s tough to tell. You see each of your own failures, and mostly hear about other people’s successes. So it’s easy to imagine that you’re worse than average at doing stuff.

But we all forget to send emails and to look things up. We all fail to keep in touch with friends who we meant to stay connected with. It may seem too late to send a greeting, or to finish something you said you’d get done. It may seem like you just haven’t done enough yet. At any rate, it would feel embarrassing to send it now.

Fortunately though, these rare reminders about how bad we all are can teach us a powerful lesson, which is that late, half-baked crap is more than people expect.

All you need to do is beat the world’s low expectations

The easy way out of these situations is to not do anything. Just put it down and don’t think about it. As long as it’s not work, taxes, or health stuff, the problem will usually just go away. So that’s what most of us do. Nothing. And that’s most of us expect, too. Nothing.

But nothing is only what it looks like. Most people actually do get started. We do take the first steps toward finishing stuff. Maybe just a bit late, or something makes us put it aside incomplete. Then life happens, time passes, and embarrassment keeps us silent. So it looks like nothing got done at all.

So no one expects anything of you. Which is great news!

Because you probably have a pile of late, half-baked crap lying around. Stuff you meant to do a long time ago, or that just needs a little more work. You occasionally remember it and cringe, thinking it’s too late now. False. It’s time to send it off — just apologise if you’ve gotta. Set aside the prideful embarrassment for a moment and think about this:

They aren’t expecting anything from you — or anyone

If it’s so late that you feel embarrassed about sending it, the person has probably forgotten about it, or moved on. They’re not expecting anything now, so this is unexpected gravy. You may just make their day. They won’t be angry that you’re late, or didn’t do enough, they’ll be glad that you did something. The vast majority of people can’t manage that. Your late, half-baked crap may even amaze them.

No matter what, it’s better than nothing.

You probably don’t need REST

Let’s start with the following: this is a post about a part of REST called HATEOAS, and why your API probably doesn’t need it. If you don’t know what that is — especially if you also feel that you do know what REST is — or, you think your API should implement HATEOAS, this post lies in a subset of {informative, angering, boring} for you.

I ran across Zapier’s When REST Gets Messy post, and among other good advice, it had a buried gem regarding HATEOAS:

I’m also not convinced that it makes sense for an internally consumed API.

…why is this a gem? And what on earth is this HATEOAS acronym we’ve seen 3 times in two paragraphs? Read on to find out.

HATEOAS (pronounce it however you like — I use both hate-ee-ohs and hate-yo-ass) is one requirement of a RESTful API. The requirement is that a RESTful API should use Hypertext As The Engine Of Application State.

Acronym city. So what on earth does that mean?

Roughly this: each response from your API should include a representation of all validly reachable states. Probably as a set of URLs describing what can be done next — kinda like a dynamic site map, but in an API.

HATEOAS is kind of obscure

Most people don’t know what HATEOAS is. Many have never even heard of it — including a surprising number of people who claim to have written RESTful APIs. This is because it’s a bit of a complex topic, and can only be addressed after getting people to grok some more basic requirements of REST. Also… there’s no standard way to implement it — but I’ll cover that nit in a bit.

Most people learn about REST by reading a tutorial or article, and unfortunately end up with the following understanding of it:

  1. Use HTTP verbs —  at least GET, POST, PUT, and DELETE (poor PATCH and HEAD get no love)
  2. Your database tables (e.g. “posts”, “comments”) are “Resources”, and you should expose an endpoint for each one. The HTTP verbs map well to CRUD (Create is POST, Read is GET, Update is PUT, Delete .. DELETE) actions on them.

Those are steps on the way to a RESTful web API, but it also needs some HATEOAS sauce. Whether HATEOAS is left out of articles or people just stop reading before getting to it is unknown to me, but they frequently seem to miss it.

The two points above roughly define a resource oriented API. And you know what? That is good enough for most applications.

Resource oriented is good enough

Once again, it isn’t REST, but it’s good enough for most APIs. Writers of APIs often just desire a way to organize their data and their interaction with it. A resource-oriented API provides exactly that. They’re also using HTTP verbs, so web crawlers can’t mess up their data with errant GET requests. I think this is a big win.

But… shouldn’t they do all of REST? What exactly would HATEOAS gain them? (let alone that there are other components to REST) Here is my thesis: HATEOAS is useful for solving a problem that is more complex than what most API implementers have to deal with.

As I said before, HATEOAS basically gives a list of all the links available to you. If you’re writing both an API and its client or client library, you really don’t need that. You already know what links are available. If they change, it’s you who changed them, and you who will make corresponding changes in your API-consuming clients. Most of us are not writing APIs for public consumption, and if we are, we’re writing fairly stable, versioned APIs that are typically accessed via libraries, by purpose-written integrations or apps.

Now we see why that sentence from When REST Gets Messy is a gem. An internally consumed API is a simpler problem than the one HATEOAS solves.

What problem does HATEOAS solve?

HATEOAS lets you do something truly cool: with it, you could write a client that has no knowledge of an API aside from how to read its site-map. A business with several teams that all make APIs (internal or external) might need this, because any client might have to speak to many different APIs. The business has a choice:

  1. The client can know every detail of every API,
  2. Every API can work the same way, or
  3. Every API can be described the same way.

The last choice is obviously a very good idea; it’s like a meta-API. Everyone agrees on one way for the APIs to describe themselves, and then one client library can know how to parse that description format.

It’s now okay for the APIs to be totally unstable. Clients are not constantly needing updates, because they aren’t tied to a particular data representation or quirk. The HATEOAS representation gives enough information for the client to figure out what it needs. Google or Yahoo or Amazon are the type of place for whom this is a real problem.

It’s sort of the magical dream that one day, all APIs everywhere will be self-describing, and all clients will know how to parse the descriptions and figure everything out for themselves. Then we can stop being as concerned about API stability and versions, and writing clients or fixing API bugs would conceivably be way easier.

Sounds great. Why not just implement it and go full REST?

Unfortunately, HATEOAS has some problems. The worst of which is that you have to come up with your own way to describe your API, because REST just tells you to do it, not how to do it. To my knowledge, there’s no known good way to describe all possible APIs yet — so you’re on the hook for that.

Then the smaller problem: adding in self-descriptions takes extra work. Just writing a client that parses them can be a lot of work.

There are some attempts at making a standard, but we’ve all seen that xkcd. If one takes off, people could be writing general purpose API consumers, and that would be cool. For now though, there are a million ways to describe your API, so put yourself in the shoes of someone writing a client. You have the following choice:

  1. Pay close attention to the HATEOAS implementation of the API and write a consumer to parse it and figure out what to do dynamically.
  2. Pay close attention to the Resource implementation of the API and write a consumer to interact with it, and update it when needed.

The latter option is much easier up-front. So consumers rarely use HATEOAS, even if it’s present. Now, put yourself back in the shoes of an API writer: is it worth spending the time to a) come up with a description format and then b) implement it? In practice, most API writers end up in one of the following situations:

  1. Spend extra time to write a beautiful, fully RESTful API and have no one really use the HATEOAS components.
  2. Bolt a crappy implementation of HATEOAS on to your resource-oriented API and have no one really use it.
  3. Actively choose not to provide a HATEOAS representation because you know no one would use it anyway.
  4. Blissfully remain ignorant of the fact that your non-HATEOAS API is not a RESTful API.

And some rare organizations fall into this category:

5. Spend extra time to write a beautiful, fully RESTful API that you and your
customers use, because your constellation of APIs are constantly in flux.

You probably don’t need REST.

It’s a fun mental game to play, and it can be smart to include links in your API responses to ease a client’s work. Fully representing your application’s state in every response is almost always overkill — so until we have solved how to do HATEOAS everywhere, a resource-oriented API is good enough.

Let’s figure out mailing lists.

I don’t really use mailing lists, but I have often seen references to their utility.

This is a bit of an exercise in trialling a research methodology out. I’m interested in making a program to help me do this, but I’m going to try just writing all the stuff I think I want to save in a database down in a blogpost, and see how cumbersome it is. I already do a lot of this in my head, but I sometimes decide not to bother, because no one can see what I have only done in my head.

Topic: Mailing Lists

Questions:

  1. I don’t know what program (if any) is required to use (read: participate in) them effectively, or at all. I don’t particularly like using gmail these days — I have thousands of unread messages.
  2. I don’t know what a good set of mailing lists would be to subscribe to.
  3. I don’t know how to find them.
  4. I don’t know what etiquette prevails on them
  5. I don’t know how multiple threads are kept running or how to not let them overwhelm me. (in other words: how to manage them)
  6. I don’t know what I don’t know.

Links:

  1. From a google search for “using mailing lists”:
    1. http://www.livinginternet.com/l/lu.htm
    2. http://manuals.kerio.com/kms/en/sect-mlistusage.html
    3. http://manuals.kerio.com/kms/en/sect-mlistusage.html
    4. http://en.wikipedia.org/wiki/Mailing_list
    5. http://programmers.stackexchange.com/questions/71148/why-do-programmers-still-use-mailing-lists
  2. From a google search for “reading mailing lists”:
    1. http://softwarerecs.stackexchange.com/questions/3213/client-for-reading-several-big-mailing-lists
    2. http://stackoverflow.com/questions/351318/best-practices-to-follow-read-large-mailing-lists

Know:

  • I know that mailing lists have smart people discussing interesting things on them
  • I know that I could just subscribe to a given one (probably by emailing it) and start getting messages.
  • I know that I would probably get a lot of messages if I subscribed to more than a few active lists.

Hmmmm. Neat. So, in flushing out the things I know and the things I think I need to figure out, an important truth has been discovered: I already know how to use mailing lists sufficiently well. I mean, goodness, I use them at work every day. For some reason, mailing lists in the wild just seemed like a different beast to me.

My questions are more about “how to be a power user of these” than “how do they even work” — but even there, I think I have a fair grasp. For example:

  • I know that etiquette will be somewhat variable between lists and the kind of thing you just have to feel out.
  • I know that I can find lists by going to programming language or project sites and finding their information out, or asking in an associated IRC channel if one exists.

I do not know how to manage them or what I don’t know.

Want To Know

  • I want to know how to manage them effectively.
  • I want to know what I don’t know, haha. I’ll keep an open mind on this.

Learning

This shift in ‘what I want to know’ suggests I should perhaps issue a new google query, ‘how to manage mailing lists’ rather than ‘how to use mailing lists’. A google search for that shows why it is a bad idea: most people who search that are looking to manage a list, not manage their interaction with several lists. After a few more trial queries, I landed on the simple “reading mailing lists”, which gave me links 2A and 2B.

2B is a little worthless and short, but 2A looks like the kind of goldmine I was looking for, and it’s also telling me something I’d wondered: a specific client, like mutt, may be my best option.

Analysis

I think I can’t actually know super practically what managing them entails until I start trying, so maybe I should just subscribe to a bunch and see how it goes. If gmail fails me, and I can’t find anything out there, I’ll can build something that solves my problems. The suggestion of trying mutt is likely where I’ll turn to first once I run into a problem, but it might be nice not to have to boot a vm to read mail (I use windows on my home laptop.)

Thanks for bearing with me through this slightly silly experiment.

It bothers me that no one updated the page

Just looking at aaronsw’s stuff. “If I get hit by a truck…”, from 2002, is a neat read. I don’t have much comment (I certainly don’t have plans for what should be done in the event that I am hit by a truck, however proverbial) — I just kind of wanted to say somewhere, to somebody (even you, Dear Reader) that it’s upsetting to me that at the bottom of that page the footer says “I’m not dead yet!”.

The guy’s dead. And he directly asked that the page’s footer be updated to note what’s happened if something happened. That’s not the kind of thing that you don’t do.

Maybe no one has permissions? I hope that’s it.

Update: I reached out to the person listed as being responsible, and he was extremely accommodating to some random internet passerby asking about a fairly personal thing. His response made sense — estates are complex things to deal with, and there are often both a lot of things to do and a lot of things keeping them from being done.

Installing pygame on Max OS X 10.8.5 Mountain Lion

I first tried to follow this guide, which did install pygame but left it spitting this kind of error when I tried to actually use it:

Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Error (1000) creating CGSWindow on line 259'

That sucked, so here is an alternative.

I assume that you already have a 64-bit version of Python 2.7.2+ along with pip and virtualenv, and the XCode Command Line Tools.

First, install XQuartz. It is an X11 server, which SDL needs, and which is no longer in OSX. The official version is here, but that site is flaky, so I downloaded it from CNET(direct download link). After running that dmg, you should log out and back in, then start the x server (e.g. cmd+space, ‘xquartz’, enter). If it shows up in your dock, you’re good.

Next, we install SDL.  Download the OSX 10.5 runtime library dmg from libsdl.org and run it. Inside, just find the SDL.framework folder and copy it into /Library/Frameworks — you’ll probably need to enter your password.

Next set the environment variables needed to perform the compilation:

export CC='/usr/bin/gcc'
export CFLAGS='-isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk -I/opt/X11/include -arch i386 -arch x86_64'
export LDFLAGS='-arch i386 -arch x86_64'
export ARCHFLAGS='-arch i386 -arch x86_64'

(thanks to james friend for writing the guide I linked above, which this is directly lifted from). The effects of these lines will only last as long as your terminal is open, so don’t close it until you get things working.

You can now make a virtualenv, activate it, and install pygame:

virtualenv game-env
. game-env/bin/activate
pip install pygame

It should say something like:


Downloading/unpacking pygame
  Downloading pygame-1.9.1release.tar.gz (2.1MB): 2.1MB downloaded
  Running setup.py egg_info for package pygame

    WARNING, No "Setup" File Exists, Running "config.py"
    Using Darwin configuration...

    Hunting dependencies...
    Framework SDL found
    Framework SDL_ttf not found
    Framework SDL_image not found
    Framework SDL_mixer not found
    Framework smpeg not found
    PNG     : found
    JPEG    : found
    SCRAP   : not found
    PORTMIDI: not found
    Framework CoreMidi found

    If you get compiler errors during install, doublecheck
    the compiler flags in the "Setup" file.

    Continuing With "setup.py"

As long as “SDL” was found, you’re in a good spot.

If you get an error that looks like this at the very end:

/Library/Frameworks/SDL.framework/Versions/Current/Headers/SDL_syswm.h:58:10: fatal error: 'X11/Xlib.h' file not found

#include <X11/Xlib.h>

         ^

1 error generated.

error: command 'clang' failed with exit status 1

then you’ve probably got to paste in the export lines listed above.

If you get an error that looks like this at the end:

src/_numericsurfarray.c:1100: warning: implicit declaration of function ‘SDL_FreeSurface’

lipo: can't open input file: /var/folders/5y/shcnnkg970n53k9bvdvngw8w0000gp/T//cc1KHqyH.out (No such file or directory)

error: command '/usr/bin/gcc' failed with exit status 1

with something like this near the top:

Installing collected packages: pygame
  Running setup.py install for pygame
    building 'pygame._numericsurfarray' extension
    /usr/bin/gcc -DNDEBUG -g -Os -Wall -Wstrict-prototypes -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk -I/opt/X11/include -pipe -arch i386 -arch x86_64 -I/NEED_INC_PATH_FIX -I/System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c src/_numericsurfarray.c -o build/temp.macosx-10.8-intel-2.7/src/_numericsurfarray.o
    In file included from src/_numericsurfarray.c:23:
    src/pygame.h:106:17: error: SDL.h: No such file or directory

then you’ll need to double check that you installed SDL properly. If you try this over and over and it doesn’t work, you might try to play with the export lines to find your copy of SDL, or follow the guide I linked at the start, which uses homebrew to install SDL.

If things are messed up, you can always just delete the SDL.framework folder, delete your virtual environment, and start again.

If things look good, try running a sample program like the following:

>>> import pygame
>>> pygame.init()
(4, 0)
>>> pygame.display.set_mode((640, 480))
<Surface(640x480x32 SW)>
>>> pygame.quit()
>>> ^D

A screen should appear, and execution should finish cleanly. If that’s the case, you can make games with pygame. 🙂