dd to the rescue

Recently, I needed to rescue a DVD (legally owned!) with read errors by making a copy. In Linux, the “device dump” or dd command can be useful, because it can read past disk errors and thus salvage whatever is intact. For example,

dd if=/dev/sr0 of=rescued_dvd.iso bs=2048 conv=noerror,notrunc iflag=nonblock

tells dd to perform low-level read (blocksize 2048bytes) from the raw device, i.e., /dev/sr0, to write whatever was salvageable to the file “rescued_dvd.iso” and to skip over errors (most importantly, the conv=noerror parameter). Fortunately, I realized that there is something even better, a neat utility called ddrescue. In principle, ddrescue performs the same low-level read, but it is optimized for failing hard drives — and this prompted this post.

I have rescued my share of hard disks, often for colleagues who came to me as a last resort. There are several tricks with which data from a failing hard drive can be recovered — even in cases that seem hopeless. Typically, Windows gives up first, claiming a nonexistent or unformatted disk (hint: don’t accept the offer to format the drive!). Here are some steps that I found useful to follow:

Turn off the computer. Don’t even bother shutting down. The write operation is more critical than the read operation, and any attempt to write to a failing hard disk makes matters worse. And Windows writes a lot to any drive during boot and shutdown.

Attach the drive to a Linux computer. Some computers have a hot-swap e-SATA connector. An external SATA-to-USB cradle is useful, too. Worst case, attach it internally (with power down) and boot it up. DO NOT ALLOW THE O/S MOUNT THE DISK!

A first attempt can be made by mounting the drive read-only (!) and trying to read as many files as possible. Often, 80 to 90% of the data can be recovered in this fashion.

Next, ddrescue comes into play. Since the low-level device is accessed, the drive does not need to be mounted. Let’s assume the drive maps to /dev/sdd. In the simplest form

ddrescue /dev/sdd disk_rescue.img

starts ddrescue and tries to image the entire drive to the file disk_rescue.img. Note that this is a raw read operation, and the image file will be as large as the entire drive capacity. For example, if you image a 1TB hard disk, you need to have 1TB free space — if necessary, use a separate hard disk.

The interesting feature of ddrescue is that it attempts to read good sectors first (to salvage as much as possible, assuming that the drive keeps degrading), then repeatedly reads the bad sectors — often, repeated reads are rewarded with a random success. This process can take days, literally. When the process is finished, mount the image file and examine what you salvaged.

DISCLAIMER: This article is not intended as advice, and the author assumes no warranty, either explicit or implied. If you have critical data, consult a professional data recovery service.

 

 

Raspberry Pi: Auto-starting a control system at boot time

Micro-sized computers, such as the Raspberry Pi, become increasingly popular as high-end replacements for microcontrollers. If a control program needs to be run in the Raspberry Pi, we would often like it to start automatically at power-up (much like a microcontroller would behave), or, more precisely, at boot time.

The information in this post can be found in various places on the Internet. It is nothing new, and I consider this to be a sort of yellow sticky note for myself — plus everybody else who might find this interesting.

Typically, the Raspberry Pi boots up and presents a login screen, either a text-mode console (TTY) or an X-window-based GUI. In either case, we need to manipulate the boot process. I assume that you (the reader) are familiar with text editors, the command-line interface, and with simple Linux behavior, such as the fact that you need to have root rights to edit files in /etc (and, for that matter, know what /etc means).

Step 1: Create the user account under which the program will run

You probably don’t want to run your application program with root privileges. Therefore, let’s create a dummy user under whose account the application will run. Let’s call that user “app”. Since we need that user to have a valid login, you also need to create a strong password. You know… if it looks like a Perl statement, you are doing something right. Note: After you added the user (e.g., with “adduser”), you may need to add this user to several groups to provide access to I/O functions. Relevant groups could be, for example, plugdev, dialout, or gpio.

Next, install the application program and verify that it works. You do this by logging in under the new user “app” and starting the program as you would normally do.

Step 2: Auto-login the new user

Now, we will modify the boot table so that the new user is logged in automatically whenever the Pi is booted up. The relevant file for this purpose is /etc/inittab. Normally, this file spawns six TTY console logins (getty) to allow normal login. Choose one console (TTY1 is the one that shows up on boot, but any other console works, too). Let’s assume we log in the user on TTY2. Fairly far down in /etc/inittab, you’ll find a block that looks like this:

1:2345:respawn:/sbin/getty 38400 tty1
2:23:respawn:/sbin/getty 38400 tty2
3:23:respawn:/sbin/getty 38400 tty3
4:23:respawn:/sbin/getty 38400 tty4
5:23:respawn:/sbin/getty 38400 tty5
6:23:respawn:/sbin/getty 38400 tty6

The first column is the TTY number. The second is the runlevel under which this TTY is activated. The third column is the action (respawn) and the fourth is the command to respawn. We now change the second line to

2:23:respawn:/bin/login -f app tty2 < /dev/tty2 > /dev/tty2 2>&1

Instead of starting a getty on TTY2, we now start a login (the -f flag specifies the user name). Also, we redirect standard input and output from/to TTY2 (i.e., the device /dev/tty2). Note also that we don’t really need “app”‘s password here. The password is useful when you need to manually login to troubleshoot.

If you reboot the system now, you should see that user “app” is automatically logged in on TTY2. Neat.

Step 3: Auto-start the application when “app” logs in

There is a file in each user’s home directory, named .profile, which is executed by the login shell. Note: If the login shell is bash, make sure that neither .bash_profile nor .bash_login exist, because those override .profile.

Complete this step by editing /home/app/.profile and appending the command to start your application. This is what it could look like towards the end of .profile:

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
PATH="$HOME/bin:$PATH"
fi

# Autostart the control program

$HOME/start_my_control.sh

The actual start of the application is now performed in a script called “start_my_control.sh”, and you can use that script file to set any environment variables, provide command-line arguments etc. as needed.

What to do when the program has a GUI and requires X window?

Disable the automatic start of the X window system on TTY7 and instead use “startx” to start the X window system from the user’s login in Step 3. Then use the window manager’s autostart feature, which depends on the window manager (xfce? lightdm?) you installed.

Why Some E-mail May Not Reach Me, And What To Do About It

As you might be aware, I try to respond to e-mail in a timely fashion. Usually, if I have not responded within a day, something may have gone wrong. After all, some e-mails do disappear in write-only memory.

How does this happen? The problem is multi-layered. I receive over one hundred e-mail messages daily. If I were to look at each of them, I’d have to pay attention to an e-mail every six minutes in average over a 10-hour workday. Obviously, this situation is untenable, especially since I am not a good multi-tasker.

For this reason, I rely heavily on e-mail filters. I handle my e-mail locally with a POP3 client instead of a webmail client. My local mail client first removes e-mails from a list of known spammers, then sorts the remaining e-mail in mailboxes that correspond to tiers. This arrangement could work well if it wasn’t for a third party — Microsoft.

UGA has outsourced their mail service to Microsoft, and e-mail is handled by Microsoft’s Office365. This has several consequences:

  • The webmail client of Office 365 is completely unusable for me, because I use a script blocker for security reasons. However, Microsoft (unlike other freemail providers, such as Gmail) offers no fallback noscript version. This means that I would have to enable scripts globally just for the “privilege” of accessing Office365. Won’t happen.
  • Since all filtering is performed locally, I do not need Microsoft’s filters, and I disabled them. However, it appears that Microsoft (“here is where you’ll go today”) has decided that I need my filters and regularly re-enables them.
  • Since I do not access the Web interface, I do not have access to Microsoft’s internal mail boxes, such as Spam or Trash. Once an e-mail was classified as Spam or Trash by Office365, it is gone for me, and becomes invisible for my e-mail client.

My local e-mail client performs additional filtering. However, if the filter classifies an e-mail as Spam, it will at least be in my local Trash folder and thus recoverable.

Some criteria that cause an e-mail to either be deleted or dropped in priority are:

  • Sending e-mail through known spammers. Any e-mail received from known spam companies (such as Constant Contact, Vertical Response, Mail Engine, B2B Mail and several dozen others) will be deleted without warning or trace. These spammers do not respect opting out and cannot seen as legitimate for this reason.
  • HTML only. Spammers frequently use HTML messages, and HTML-only messages are treated as suspicious. Consider sending plain-text e-mails or using the compliant setting where any HTML e-mail also carries the plain-text body.
  • Short text, large attachment. Often, spammers disguise their message in attached files (images or Word documents), and messages with a low text-to-attachment ratio are also suspicious. Consider embedding your message in the main e-mail body instead of using a Word attachment.
  • Listservs. Those are the bane of today’s e-mail communications, and my e-mail client relegates all listserv messages to Tier-2. If you really want to send me a message, consider sending it directly, not via listserv.
  • FYI. Any message that begins with FYI is probably not important and is relegated to Tier-2.
  • Missing name in “To:” field. Here is the real doozy. I found that 90% of all my e-mail problems are solved with one simple filter rule: If my name is not in the “To:” field, it is not important and relegated to Tier-2.

Essentially, this gives me a three-tiered system: E-mails from known spammers and those recognized as spam are sent to the trash bin. Those that meet the “not overly important” criteria are sent to Tier-2, and I may glance at them if I find the time. Everything else goes to Tier-1 and receives a proper response. With this system, I am down to perhaps 10 out of my 100 daily messages that deserve my attention.

If I missed your e-mail, my apologies. Chances are fifty-fifty that it was either “eaten” by Office365 or relegated to Tier-2 on my local system. In the latter case, it is easy to fix it and elicit Tier-1 priority.

Thanks for understanding.

Amazing math typesetting with MathJax

By accident, I stumbled on something interesting. You are probably aware that I write my books and most of the class handouts with LaTeX. Equations are particularly convenient, because LaTeX typesets equations just… right. Something that I can’t do in, say, html. It would appear, though, that others already solved this problem: A script that lets you remotely render a LaTeX equation — MathJax.

This means, you type in your html code (or post) something like

F(s) = \int_0^\infty f(t) e^{-st} dt

and what you get is:

\[F(s) = \int_0^\infty f(t) e^{-st} dt \]

Not bad, actually.

A word about Word

Many of you send me homeworks by e-mail. This is OK. We’ll save paper. You don’t print it, and I won’t print it. I’ll grade it on-screen and provide you with the result, but no detailed comments.

However, some of the attachments are doc or docx documents (Microsoft Word). Many of you probably already noticed that Word is highly incompatible with other versions of Word and other platforms (e.g., Mac OS), not to mention other programs such as OpenOffice or Libreoffice. The new docx format is even worse in its compatibility issues than the older doc format.

The best way to avoid the daily frustration associated with Word is not to use it, but this is your choice.I do not use Microsoft Word, nor do I own a copy. Actually, I do not use Windows at all any longer.

I got to a point where I feel that it is the sender’s responsibility to make sure I can open the document sent to me. If you send me a docx file, I will do my best to read it. However, if parts get lost in the conversion process, I cannot help it. If vital parts that earn a score get lost this way, I cannot help it.

Many formats, first and foremost PDF, can be opened in any operating system. Plain html is surprisingly versatile. RTF is a viable alternative, it can even handle tables and figures.

So, if you send me a file, please make sure I can read it.

 

Interactive discussion threads for homeworks!

This is an experiment. You know what I keep saying in class: Ask your question, because if you have that question, others likely have similar questions.

I would like to try something new: Instead of responding to each question individually (in the office or by e-mail), I would like you to post your question or your discussion items here, under the general discussion page or under a specific homework page. I’ll try and respond as I would to an e-mail.

If you have a question, you might want to look at the homework thread. Maybe somebody has already asked a similar question? Or maybe you’d want to try and answer a question from another student — you could actually test your knowledge that way.

Or you have a comment or an interesting web site that you’d like to point out. Just post it under the general class page.

I am looking forward to starting this experiment, and I hope that you provide me with constant feedback and critique,  and with suggestions how to further improve the communication between professor and students and between students in a class.