Jan van den Berg

j11g.com

Tag: bash

  • I don’t understand terminals, shells and SSH

    Confession time: I don’t fully understand how terminals, shells and SSH really work (and my guess is you don’t either). And I don’t mean the cryptography behind SSH. I mean how SSH and the terminal — and the shell for that matter — interact with one another.

    I recently realized that even though I’ve been daily remotely logging into Linux systems for all of my adult life (and type in the shell and Vim) I didn’t really grasp how these things actually work.

    Of course I conceptually know what a (virtual) terminal is (entering input and displaying output) and what the shell is for (the interpreter). And SSH is the remote login protocol, right? (Or is SSH a pseudoterminal inside another pseudoterminal, who’s to say)?

    The distinction between these three elements is a bit fuzzy and I do not have a clear concept of it in my head. The test being: could I draw it on a whiteboard? Or: could I explain it to a novice? The answer is probably: not really.

    So I went on a bender and found these four (well-known) links that explain things like tty, rawmode, ptms/ptx, pseudoterminals and more.

    This post functions as a bookmark placeholder. I will add more links when I find them.

    There’s lots of information here if you’re interested. And of course: you mostly don’t actually need to know any of these things to do your work — we are all forever standing on the shoulders of giants. But I *want* to understand these things. And I think I understand them a little bit better now. Maybe you will as well.

  • Use find (1) as a quick and dirty duplicate file finder

    Run the following two commands in bash to get a listing of all duplicate files (from a directory or location). This can help you clean out duplicate files that sometimes accumulate over time.

    The first command uses find to print all files (and specific attributes) from a specific location to a file, prefixing the size of the file in the name. This way all files with the same filename and same size can be grouped together. Which is usually a strong indicator that files are similar.

    When you run the second command you will get a sorted list of all actual duplicates, grouped together. This way, you can quickly pick out similar files and manually choose which ones to keep or delete.

    find . -type f -printf "%s-%f\t %f %c\t %p\n" >> /tmp/findcmd

for i in `sort -n /tmp/findcmd|awk '{print $1}'|uniq -cd|sort -n|awk '{print $2}'`; do grep $i /tmp/findcmd; done

    The output will look something like this, you can instantly tell which files are duplicates, based on size, name and/or timestamp.

    1067761-P4270521.JPG     P4270521.JPG Wed Apr 27 18:05:04.0000000000 2011        ./Backups Laptops/Ri-janne/2011 Diversen
1067761-P4270521.JPG     P4270521.JPG Wed Apr 27 18:05:04.0000000000 2011        ./Backups Laptops/Ri-janne/2011 camera
1067898-IMG_3418.JPG     IMG_3418.JPG Thu Aug 28 20:08:28.0000000000 2008        ./Piks/2008/Vakantie USA 2008/Dag 7 Louisville Shopping
1067898-IMG_3418.JPG     IMG_3418.JPG Thu Aug 28 19:08:28.0000000000 2008        ./Backups Laptops/Ri-janne/2008 USA
1067969-P9180184.JPG     P9180184.JPG Sat Sep 18 17:45:52.0000000000 2010        ./Backups Laptops/Ri-janne/2010 Diversen
1067969-P9180184.JPG     P9180184.JPG Sat Sep 18 17:45:52.0000000000 2010        ./Backups Laptops/Ri-janne/2010 uitzoeken
1068244-100_2962.jpg     100_2962.jpg Thu Jul 17 18:18:52.0000000000 2008        ./.Trash-1000/files/Mijn afbeeldingen/Italia 09/Greece '08
1068244-100_2962.jpg     100_2962.jpg Thu Jul 17 18:18:52.0000000000 2008        ./Backups Laptops/Jan/Mijn documenten/Mathea/Mijn afbeeldingen/Italia 09/Greece '08
1068284-DSC_7640.JPG     DSC_7640.JPG Sat Apr 26 14:47:58.0000000000 2014        ./Piks/2014/20140426 KDag
1068284-DSC_7640.JPG     DSC_7640.JPG Tue Apr 29 21:56:54.0000000000 2014        ./Piks/2014/20140426 Koningsdag

  • Create a Chrome bookmark html file to import list of URLs

    I recently switched RSS providers and I could only extract my saved posts as a list of URLs. So I thought I’d add these to a bookmark folder in Chrome. However, Chrome bookmark import only accepts a specifically formatted .html file.

    So if you have a file with all your urls, name this file ‘url.txt’ and run this script to create a .html file that you can import in Chrome (hat-tip to GeoffreyPlitt).

    #!/bin/bash
#
# Run this script on a file named urls.txt with all your URLs and pipe the output to an HTML file.
# Example: ./convert_url_file.sh > bookmarks.html

echo "<!DOCTYPE NETSCAPE-Bookmark-file-1>"
echo '<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">'
echo '<TITLE>Bookmarks</TITLE>'
echo '<H1>Bookmarks</H1>'
echo '<DL><p>'
  cat urls.txt |
  while read L; do
    echo -n '    <DT><A HREF="';
        echo ''"$L"'">'"$L"'</A>';
  done
echo "</DL><p>"