Using Redis to Cache the Todoist Python API

Herding the Cats

I’ve taken a liking to Todoist for tracking my daily tasks, professional goals, and helping my managers organize our major intiatives. However, their reporting is lacking. You can use the print feature, but that’s not terribly flexible. Furthermore, I’d like to be able to have some sort of mechanism to track changes over time.

Fortunately, Todoist provides a couple different APIs to develop against. They also provide support for a Python library that wraps their REST and Sync APIs.

Using the Python API

What I want to do is build a service that provides some additional value for Todoist users, and I want to implement that using serverless technologies atop AWS Lambda. When you get into the serverless environment, you are restricted on things like storing data persistently within the serverless platform.

The Todoist Python library provides two modes of operation:

• Cache results locally to files
• No caching whatsoever

Neither of these are scalable solutions, particularly if scalablity and not being throttled are requirements.

How to fix?

I am not a Python-ista. I come from a Perl, Java and, more recently, a JavaScript background. Looking at the source for the Python API, I want to immediately re-architect the API code so that storage backends can be arbitrarily changed out depending on the desired backend.

That’s a bit involved for the way the API class is currently written. I’m also not quite comfortable to take an axe to their API and fight that battle (yet).

Without completely rewriting the Todoist Python library, how did I work around this?

One thing I did learn about is monkey patching to dynamically modify classes. With that said, I was able to fairly trivally replace two methods within the TodoistAPI class in order to use Redis as a backing store for the cached Todoist data.

Is this perfect? No! However, it does allow me to continue working on my prototype without getting bogged down in a much larger conversation…

Implementing the solution

To keep it simple, all of this code landed in the prototype script I’ve been using to explore the API and the library. One could abstract this out appropriately (and perhaps that’s yet a better position than attempting to re-architect the Todoist Python library).

Below are two methods I created. The intent was to mimic, as closely as possible, the original API code as to not disturb the original functionality within the library. This was done by replacing two specific internal calls, _read_cache() and _write_cache().

You will note that I am not doing anything fancy with Redis. I had started with delusions of grandeur and started down the path of using HSET and HGET to store hashes directly. I quickly learned that this is not straightforward, especially when dealing with types other than strings. It is possible to do all sorts of contortions to get there, either manually mapping and decoding the data returned from Redis, or using Redis 4 and the ReJSON module. One should note that (at the time of writing) AWS ElastiCache is still on Redis 3, and therefore does not support modules.

With my tail between my legs, I fell back to simply serializing the JSON completely as a string and also the sync token – mimicking exactly the way the original methods work on the file system, except instead writing that data to Redis. It works for now, but I’m waiting for the other shoe to drop as I dig further into the weeds.

Read from the Cache

## Monkey patch the TodoistAPI instance to use Redis for the caching mechanism
def _monkey_read_cache(self):
    if not self.redis:
        return

    try:
        self._update_state(json.loads(self.redis.get(self.token + '.json').decode('utf-8')))
        self.sync_token = self.redis.get(self.token + '.sync').decode('utf-8')

    except AttributeError:
        print('[WARN] - There was no data to decode (likely a cache miss).')
        return

    except:
        print("Unexpected error:", sys.exc_info()[0])
        raise

Write to the Cache

def _monkey_write_cache(self):
    if not self.redis:
        return

    self.redis.set(self.token + '.json', json.dumps(self.state, default=state_default))
    self.redis.set(self.token + '.sync', self.sync_token)

def state_default(obj):
    return obj.data

Below is where the magic happens. Using the ability to monkey patch, I insert my newly crafted methods in place of the originals. The code following this proceeds to initialize the API and use it as per the Todoist Python library docs.

Inject Methods into the Todoist API

# Inject the functions
todoist.TodoistAPI._read_cache = _monkey_read_cache
todoist.TodoistAPI._write_cache = _monkey_write_cache

# Inject a working Redis session into the TodoistAPI instance
todoist.TodoistAPI.redis = redis.StrictRedis(host="localhost", port=6379, db=0)

Rethinking the Python Library

I’m surprised they didn’t do this out of the gate. They did such a good job abstracting out all of the various object types that could come from their web service, but didn’t fully think through the caching issue. I do give them kudos for considering SOME FORM of caching, as that provides immediate relief on their backend. However, the library needs some work in order to provide scaling and protections on the “client’s” side as well. Here’s what I’m thinking:

• Abstract the caching piece out of the TodoistAPI class and implement a generic “None” caching class (aka the general interface). This interface would implement empty _read_cache() and _write_cache() methods.

• Provide a default file system caching class that would implement the current file system reading / writing that is currently embedded in the current TodoistAPI class.

• Allow the TodoistAPI class to pass in a configurable caching class. This would be something like a formal implementation of this Redis monkey patch, or other classes that implement caching with other backing stores. If no caching class is given to the API, it defaults to the file caching class so that it behaves as it does today out of the box.

I have forked the todoist-python library, but I’ve not yet committed the changes I’ve been batting around. I will do that soon.

Wrapping Up

That’s it! Making a couple slight tweaks to the existing Todoist Python library will enable you to write to Redis as a caching store. One could conceive that it would be equally trivial to implement these methods to write to other backends like DynamoDB or other platforms.

I have provided this full snippet as a Gist on Github.