foster
/ˈfɒstə/
verb
1. Encourage the development of (something, especially something desirable). “the teacher’s task is to foster learning”
TLDR: I made a personal bookshelf management web application and named it Foster and you can find it here. Here’s what I did — with gifs–, so you might build your own.
Name
I named it Foster. Because of *this* blog post — it accompanies the application, so it’s self-referential. And also, because I am currently reading David Foster Wallace‘s magnum opus Infinite Jest. And lastly, the word ‘foster’ makes a lot of sense otherwise, just read on 😉
Background
I like to read and I like to buy physical books — and keep them. Over the years I tracked both of these things in a spreadsheet. But this became unmanageable so I needed something else.
Something like Goodreads but self-hosted. So, preferably a web application where I could:
- track my reading progress
- keep track of my bookshelf
But I couldn’t find anything that fit, so I knew I probably had to roll my own. In simpler times MS Access could do this sort of thing in a heartbeat. But it’s 2019 and I wanted a web application. However I am not a web developer and certainly not a frontend developer.
But when I came across https://books.hansdezwart.nl/ I knew this was what I was looking for! So I emailed Hans. He was very kind in explaining his application was self-coded and not open-source, but he did provide some pointers. Thanks Hans! So with those tips I built my own application (front and back) from scratch. And I decided to pass the knowledge on, with this blog.
This is what the Foster frontend looks like. It’s pretty self-explanatory: I can search *my* books, track and see reading progress, track collections, click through to book details and see the activity feed (more on that later). Oh, and it’s fast! ♥
Frontend
The five different parts in the frontend are: ‘Search’, ‘Statistics’, ‘Currently reading’, ‘Collections’ and ‘Activity feed’. They are presented as Bootstrap cards. The frontend is just one index.php file with a layout of the cards. All cards (except ‘Search’) are dynamically filled with content expressed as a div class. The class content is generated by one JavaScript function per card, which in turn call a PHP file. And the PHP files just echo raw HTML.
Other than the index.php file there is one search.php file to make up the frontend. This file takes care of presenting the book details, search output, log and lists views (more on that later). So, most of what can be done and seen in the frontend is handled by the search.php file.
The frontend itself is of course nothing unique. It’s just a representation of the data. The backend is a bit more interesting!
Database
The frontend was the easy part. At least it was after I figured out the backend! I spent quite a bit of time thinking about the database design and what the backend would have to do. I thought the design for such a small application wouldn’t be too difficult. But I surprised myself with the number of changes I made to the design, to get it just right. And I wanted to get it right because:
General rule of thumb: when you start with a good design, everything else that comes after will be a lot easier.
The multiple foreign-key relations between tables (on ids etc.) are not defined in the database. I choose to do this in the code and the JOIN queries.
It’s not hard to understand the database design. And yes, the design could be a little tighter — two or three tables — but let me explain!
Log, actions and states
One of the main things I spent time thinking about are the actions and their respective states.
I figured you can do one of five things with a book (actions):
- You want a book
- You get/buy/own the book
- You start reading it
- You finish reading it
- You purge/remove/sell/give away the book
Makes sense right? You could even call it the ‘book life cycle proces‘. With one input and one output.
HOWEVER! Some books you already own without wanting them first. Or, you can read the same book more than once. Or, you can give a book to a friend, and buy it again for yourself. Or, you can finish reading a book, that you lent — from a friend or library — so it is not on your shelf anymore. All of these things happen. So actually the ‘life cycle’ actions are not a chronological fixed start-to-end tollgate process, it’s continuous and messy.
Book log
Every new action is added to the book log. In the frontend the last 25 entries to the book log are presented as the Activity feed. Every action has a timestamp when an action got logged and a date for that action. Which are two different things. So when I now add a book to my shelf that I acquired 4 years ago, the book log timestamp is now, but the date for the action is 4 years ago.
With this log I can keep track of books even after I got rid of them (because selling/purging is just one of the action for a book). This is important because this way I don’t lose the log history of a book.
Also I can add books to my wanted list even if I have owned them before (maybe I gave them away etc.). And I can start/finish reading the same book more than once. It doesn’t matter, because it is just a log entry.
Now here’s the interesting thing. With all this log information I can generate four states:
- Books I want
- Books I own
- Books I have read
- Books I had
These states are generated by specific hardcoded queries per state. They are generated on the fly by what is in the log file, and where the most recent log records prevail to decide the current status.
And with all this:
Foster will track the complete history per book and at all times represent all books I want, own, have read or have owned, at that specific moment in time.
Lists
I could have defined these actions as a list: but lists are simpler. Let me explain.
I tend to collect and read specific genres of books, e.g. music, management and computer history books. So I tend to organize books like that. These descriptions/genres are all, of course, just lists.
Some books can be three of these things at the same time: part biography, part computer history part management. So one book can be a member of more than one list.
In the Foster backend I can add or delete books to and from as many lists as I like.
I can also easily create new lists. Say: a list of books that I want for my birthday, or books that are on loan, or books that are signed by the author etc. I just add one new list to my table, and the list will be instantly available in the backend and presented in the frontend.
Collections
In the frontend the action log states and the different lists are grouped together under the Collections card. As stated the first 4 collections are populated from the log, and a book always has a last state. The others are just lists.
I can create or delete as many lists as I’d like, and it won’t affect the book log. This way I can organize my book collection far better than I could physically (a book can only have one spot on your physical shelf).
Adding books with the Bol.com API
This is where the magic happens! Bol.com — a large Dutch book retailer — has a very easy API you can use to query their book database. I use it to search and add books to my collection. With one click I can get most book details: title, ISBN (=EAN), image, description etc. And I can pull them all into my own database. Including the image, which I then copy and store locally. Like this:
Of course I can also edit book details when necessary, or just enter a book by hand without the API. Sometimes Bol.com does not carry a book.
Backend
The bol.com search API is the start page of my backend. The other important backend page is an overview of all my books. Clicking on the titles brings up an edit view of a book. But most importantly I can quickly add or delete books from lists here AND add actions (started reading, finished).
I have defined jQuery actions on the <select option> dropdown menus, which provide a popup — where I can fill in a date if necessary — and which trigger database inserts (there definitely might be some security concerns here: but the backend is not public).
Security
The frontend is open for everyone to see. I don’t mind sharing (my podcast list is also public), also because I always enjoy reading other peoples lists or recommendations. The backend is just one .htaccess password protected directory. In my first database design I had a user table with accounts/passwords etc. But the .htaccess file seemed like the easiest/quicker solution for now.
Tech
I built Foster from scratch, no Symphony/Laravel or what have you. And I am a bit annoyed surprised there is still no MS Access RAD equivalent for the web in 2019 (i.e. a all in one tool: from DB design to logic to GUI design to runtime).
I know Django does most of the backend for you , so I briefly looked at it. But for Foster I still ended up using PHP / MariaDB / Bootstrap 4 / JavaScript / jQuery. It’s a familiar and very portable stack that you can mostly just drop and run anywhere (and most answers are on StackOverflow 🤓).
I’ve thought about using SQLite, but I am very familiar with MySQL/MariaDB so that made more sense. Also I learned more about Bootstrap than I actually cared about, but that’s alright. And I wrote my first serious piece of JavaScript code ever (for the dropdown select actions). So that was fun.
All in all: I spent a few days pondering the database design in the back of my mind. And 4 evenings programming front and backend. And now I am just polishing little things: which is a lot of fun!
Further development
Right now, I still have around 200 more books from my library to catalogue correctly — that’s why some dates are set to 0000-00-00. But here are a few possible new features I am thinking about:
RSS feed for the activity log? Now that I am bulk adding books the activity feed is not so relevant, but when things settle down, who knows, people might be interested.After I wrote a draft of this blog I implemented it!- Twitter integration? Posting the log to a dedicated Twitter feed.
Adding books by scanning the barcode / ISBN with your phone camera? If I can just get the ISBN I can automate bol.com API to do the rest. Might speed things up a bit (and might be useful when you run a secondhand bookstore 😉).I created an iOS shortcut that does exactly this! It scans the book barcode/ISBN/EAN and opens the Foster edit.php URL with this ISBN number and from there I can add the book by clicking ‘Add’ (all book details are available and prefilled by the Bol.com API). It’s great!- Storing/tracking more than books? CDs, DVDs, podcasts I listened too, movies I watched etc.
- Multi-user? In the first database design there were multiple users that could access / add the books that were already in the database but still create their own log and lists. I think I could still add this to the current design.
- As you can see in the database design, there is a remarks table. I haven’t used this table. A remark is a ‘blog’ (or a short self-written review) of a book, that can be presented with the book details. This is a one-to-many relationship, because you might want to make new remarks each time you reread a book. But, I currently blog about every book I read, so the remarks might be just an embedded blog link?
Just share the source already!
“Foster looks nice. Please share the code!” No, sorry, for several reasons.
- I made Foster specifically for me. So chances that it will fit your needs are slim and you would probably still need to make changes. In this post I share my reasoning, but you should definitely try to build your own thing!
- When Foster was almost done, I learned about prepared statements (did I mention I am not a web developer?)… so I had to redo the frontend. But I haven’t redone the backend (yet): so it’s not safe from SQL injections or other pretty bad coding standards. Open sourcing it can of course generate code improvements, but it would first make my site vulnerable.
- But most importantly: Building a web application to scratch your own personal itch and learning new things can be one of the most fun and rewarding experiences you will have! And I hope this blog is useful to you, in achieving that goal.
