I’ve taken a liking to Todoist for tracking my daily tasks, professional goals, and helping my managers organize our major intiatives. However, their reporting is lacking. You can use the print feature, but that’s not terribly flexible. Furthermore, I’d like to be able to have some sort of mechanism to track changes over time.
What I want to do is build a service that provides some additional value for Todoist users, and I want to implement that using serverless technologies atop AWS Lambda. When you get into the serverless environment, you are restricted on things like storing data persistently within the serverless platform.
The Todoist Python library provides two modes of operation:
- Cache results locally to files
- No caching whatsoever
Neither of these are scalable solutions, particularly if scalablity and not being throttled are requirements.
That’s a bit involved for the way the API class is currently written. I’m also not quite comfortable to take an axe to their API and fight that battle (yet).
Without completely rewriting the Todoist Python library, how did I work around this?
One thing I did learn about is monkey patching to dynamically modify classes. With that said, I was able to fairly trivally replace two methods within the TodoistAPI class in order to use Redis as a backing store for the cached Todoist data.
Is this perfect? No! However, it does allow me to continue working on my prototype without getting bogged down in a much larger conversation…
To keep it simple, all of this code landed in the prototype script I’ve been using to explore the API and the library. One could abstract this out appropriately (and perhaps that’s yet a better position than attempting to re-architect the Todoist Python library).
Below are two methods I created. The intent was to mimic, as closely as possible, the original API code as to not disturb the original functionality within the library. This was done by replacing two specific internal calls, _read_cache() and _write_cache().
You will note that I am not doing anything fancy with Redis. I had started with delusions of grandeur and started down the path of using HSET and HGET to store hashes directly. I quickly learned that this is not straightforward, especially when dealing with types other than strings. It is possible to do all sorts of contortions to get there, either manually mapping and decoding the data returned from Redis, or using Redis 4 and the ReJSON module. One should note that (at the time of writing) AWS ElastiCache is still on Redis 3, and therefore does not support modules.
With my tail between my legs, I fell back to simply serializing the JSON completely as a string and also the sync token – mimicking exactly the way the original methods work on the file system, except instead writing that data to Redis. It works for now, but I’m waiting for the other shoe to drop as I dig further into the weeds.
Below is where the magic happens. Using the ability to monkey patch, I insert my newly crafted methods in place of the originals. The code following this proceeds to initialize the API and use it as per the Todoist Python library docs.
I’m surprised they didn’t do this out of the gate. They did such a good job abstracting out all of the various object types that could come from their web service, but didn’t fully think through the caching issue. I do give them kudos for considering SOME FORM of caching, as that provides immediate relief on their backend. However, the library needs some work in order to provide scaling and protections on the “client’s” side as well. Here’s what I’m thinking:
Abstract the caching piece out of the TodoistAPI class and implement a generic “None” caching class (aka the general interface). This interface would implement empty _read_cache() and _write_cache() methods.
Provide a default file system caching class that would implement the current file system reading / writing that is currently embedded in the current TodoistAPI class.
Allow the TodoistAPI class to pass in a configurable caching class. This would be something like a formal implementation of this Redis monkey patch, or other classes that implement caching with other backing stores. If no caching class is given to the API, it defaults to the file caching class so that it behaves as it does today out of the box.
I have forked the todoist-python library, but I’ve not yet committed the changes I’ve been batting around. I will do that soon.
That’s it! Making a couple slight tweaks to the existing Todoist Python library will enable you to write to Redis as a caching store. One could conceive that it would be equally trivial to implement these methods to write to other backends like DynamoDB or other platforms.