When I approach this kind of problem, I imagine script wizards who write a single long line of ridiculous bash, run it, and the job is done.
I am not that person. I am, however, someone who can break a job down into single steps, test them on small batches, then kick off long-running jobs while I work on the next script down the chain.
That is an efficient, effective, accurate and testable approach to the problem. It might not yield the fastest runtime, but I was happy to spend a day or two writing the scripts and a day or two to run them in parallel.
That was still going to be a LOT faster than manually checking the photos!
I can also identify problems more easily if only one requirement is addressed at a time.
The most important thing with any project is be able to reverse any action.
Everyone has learnt this the hard way. Often more than once.
I made a copy of the entire archive and put it to one side.
I identified the media files (movies and images) by extension that I wanted to keep (after expanding any zip files).
Then I removed the rest. Find is very powerful and can do a lot (efficiently) in a single command.
# find non-media files
find . -not -name "*.jpg" -not -name "*.JPG" -not -name "*.HEIC"
-not -name "*.json" -not -name "*.mp3" -not -name "*.mpg" -not -name "*.MOV"
-not -name "*.GIF" -not -name "*.PNG" -not -name "*.bmp" -not -name "*.wma"
-not -name "*.jpeg" -not -name "*.AVI" -not -name "*.jpg" -not -name "*.m4a"
-not -name "*.wmv" -not -name "*.gif" -not -name "*.png" -not -name "*.TIF"
-not -name "*.MP3" -not -name "*.mov" -not -name "*.MPG" -not -name "*.avi"
-not -name "*.3gp" -not -name "*.BMP" -not -name "*.WMV" -not -name "*.mp4"
-not -name "*.mpeg" -not -name "*.wav" -not -name "*.ogg" -not -name "*.m4v"
-not -name "*.dxf" -not -name "*.svg" -not -name "*.eps" -not -name "*.ai"
-type f > out.txt
# find non-media files and copy
find . -not -name "*.jpg" -not -name "*.JPG" -not -name "*.HEIC"
-not -name "*.json" -not -name "*.mp3" -not -name "*.mpg" -not -name "*.MOV"
-not -name "*.GIF" -not -name "*.PNG" -not -name "*.bmp" -not -name "*.wma"
-not -name "*.jpeg" -not -name "*.AVI" -not -name "*.jpg" -not -name "*.m4a"
-not -name "*.wmv" -not -name "*.gif" -not -name "*.png" -not -name "*.TIF"
-not -name "*.MP3" -not -name "*.mov" -not -name "*.MPG" -not -name "*.avi"
-not -name "*.3gp" -not -name "*.BMP" -not -name "*.WMV" -not -name "*.mp4"
-not -name "*.mpeg" -not -name "*.wav" -not -name "*.ogg" -not -name "*.m4v"
-not -name "*.dxf" -not -name "*.svg" -not -name "*.eps" -not -name "*.ai"
-type f -exec cp --parents {} /NonMediaFiles \;
# delete non-media files
find . -not -name "*.jpg" -not -name "*.JPG" -not -name "*.HEIC"
-not -name "*.json" -not -name "*.mp3" -not -name "*.mpg" -not -name "*.MOV"
-not -name "*.GIF" -not -name "*.PNG" -not -name "*.bmp" -not -name "*.wma"
-not -name "*.jpeg" -not -name "*.AVI" -not -name "*.jpg" -not -name "*.m4a"
-not -name "*.wmv" -not -name "*.gif" -not -name "*.png" -not -name "*.TIF"
-not -name "*.MP3" -not -name "*.mov" -not -name "*.MPG" -not -name "*.avi"
-not -name "*.3gp" -not -name "*.BMP" -not -name "*.WMV" -not -name "*.mp4"
-not -name "*.mpeg" -not -name "*.wav" -not -name "*.ogg" -not -name "*.m4v"
-not -name "*.dxf" -not -name "*.svg" -not -name "*.eps" -not -name "*.ai"
-type f -exec rm {} \;
After this process, I removed any directories that were left empty.
# find empty directories
find . -empty -type d > emptydirs.txt
# delete empty directories
find . -empty -type d -delete
Next we need OCR. I installed Tesseract OCR, https://tesseractocr.org/.
Depending on your linux installation, you may or may not need to also choose a language pack from about fifty.
I picked the one that ended with -en for English.
We can run tesseract like this.
# tesseract usage
tesseract inputfile outputfilename
This creates outputfilename by default as
I only want to OCR image files so I identified them with this script.
#!/bin/bash
#scango.sh
# get images, not movies, etc
find . -type f -exec sh -c '
for file do
if file --mime-type -b "$file" | grep -q "image/"; then
echo "$file"
fi
done
' sh {} + > $1
I ran the script from the top directory of the archive and got a count of the number of files I have to run OCR on (about 110,000).
~/scripts/scango.sh allfiles.lst
wc -l allfiles.lst
I created a little script to run against each image file in a list.
In testing, I discovered double-quotes were needed around $arg to handle filenames with spaces in them.
#!/bin/bash
#tessgo.sh
while read arg; do
tesseract "$arg" "$arg"
done
Then I created a few tests to estimate the length of time of the OCR process.
# testing one file with imagefiles1.txt
~/scripts/tessgo.sh < imagefiles1.lst
# testing a few files with imagefiles.txt
~/scripts/tessgo.sh < imagefiles.lst
tessgo.sh seemed to process about three files per second.
So this process would take about 10 hours.
Still better than manually checking the files.
# start main job while continuing to work on other scripts
# expect finish about 10pm
~/scripts/tessgo.sh < allfiles.lst
This job produces a txt file for every image file throughout the directory structure.
The OCR was amazing. It identified a photo as containing words even if they were just a few words on a toy box in the background of the image.
While I waited for the OCR to finish, I wrote and tested little scripts to inspect the output txt files, and remove any images that were potential documents.
#!/bin/bash
# countwords.sh
while read arg; do
wc -l "$arg"
done
#!/bin/bash
# wcountfilter.sh
minwords="$1"
#echo $minwords
while read arg; do
wcarr=($(wc "$arg"))
words=${wcarr[1]}
if [ $words -ge $minwords ]; then
echo $arg
fi
done
#!/bin/bash
# renamefiles.sh
#echo $minwords
while read arg; do
fname=$arg
imgfname=${fname:0:-4}
echo $imgfname
done
#!/bin/bash
# delimgfiles.sh
while read arg; do
rm "$arg"
done
# check word count in each file
~/scripts/countwords.sh < textfiles.lst
# test parameter + input file
~/scripts/wcountfilter.sh 15 < textfiles.lst
# if more than 15-25, write file to deltextfiles.lst
~/scripts/wcountfilter.sh 25 < textfiles.lst > deltextfiles.lst
# write list of files with .txt removed
~/scripts/renamefiles.sh < deltextfiles.lst > delimgfiles.lst
# delete image files after spot checking
~/scripts/delimgfiles.sh < delimgfiles.lst
# remove all text files
find . -name "*.txt" -delete
With over 100,000 photos I could afford to lose a few rather than let an important document slip through. I set my threshold word count appropriately after testing.
And there it is. A sanitized directory tree with over 100,000 image files, plus many thousand more home video files.
I find it interesting when the structures of interactive fiction can be reduced to a number of repeating patterns. These are likely to be structures that writers are able to adhere to, and players are likely to be able to make sense of.
As a workmate once said to me, “The difference between real life and fiction is that fiction has to make sense.”
]]>This is just a bookmark for myself. I’ll report back if it ever actually SAVES me.
Until then, it is no better than any other backup system ;)
]]>“Away in a manger, no crib for a bed,” he sang to no one. The sofa was too short. “But at least I am not weightless.”
He opened his eyes and sat up, no harness needed. The Agency had decided that putting centrifugal spin on their craft was cheaper than developing technologies to contain the effects of zero-gravity nausea. It contained the vomiting. Less food and liquid was required per person per day. Less waste disposal and clever air filter design was demanded. It also contained a blackmailing media. No more bribes were necessary to hide the sordid unromantic details of space travel from a fickle public.
“Well, that wouldn’t be a cost now, anyway,” Heinrick chuckled to himself.
The space program was a delicate balancing act of public opinion and government support. Sensing this, the press made away with a sizeable slice of taxpayers’ involuntary contributions to the space program. That was until Heinrick Jones suggested to the Agency that they should focus on developing technology that reduced the overall cost of space travel.
Impossibly, he had also improved the other side of the Agency’s accounting ledger. The Agency was making sales. Television companies climbed over each other to host pay-per-view television events of space flights. And the public was paying in droves. The space program was liquid.
“The cattle are lowing, the baby awakes” he hummed as he moved to his teak desk and sat in a comfortable leather chair. Irony heaped upon irony. Public opinion insisted the interior of spacecraft looked as much as possible like a room in a normal house on Earth. The Viewers wanted something familiar. Not like the cells of missions past. There were no windows, though. The spinning stars would drive anyone mad. But that was a small price to pay for spinning the whole ship. A faux window frame was attached to one wall surrounding a lovely photographed poster of the stars - taken from Earth.
A clock on the desk read one hour and counted down. Time for his shot at fame. Time for his shot of the latest drug developed by the Agency. Opening the top desk drawer, he withdrew a syringe and injected himself with courage.
“But little Lord Jesus, no crying he makes.”
Eventually the viewers would tire of this schtick and demand new entertainment. But for now, serious space research was being funded. A rushing sound in his ears like the sound of a long wave breaking along a shore was accompanied by a relaxing numbness in his mind and well-being in his body. He sat and waited.
A red light came on above the desk. The show began. Millions tensed.
Heinrick stood up and walked to the fake window. He placed his hands on it and could hear a sizzling sound as smoke rose from around his fingers and palms. He unpeeled his hands, looked at them, and smiled. Turning around, he held his hands up to the camera. Oozing blisters waved across space to his captive audience. Walking to the other side of the room, he pressed his face to the wall and waited for a satisfying popping sound and half-blindness. It did not take long. Facing the camera again, he danced a little. He sat on the sofa and kicked off his shoes. He stood and walked over to the desk, each step leaving a burning, smoking mark of flesh stuck to the frying pan floor. He slouched on the chair, and passed out.
Soon after, his skin began to blister and melt. The window poster ignited. His hair burst into flame. The desk caught fire. Flames ate the room. The walls melted, and the last brief camera shot was of a huge sun.
“And take us to heaven, to live with Thee there.”
]]>jekyll serve
It also automatically updates when files change.
Now I needed an efficient way to upload the website to my host.
Difficulty: no rsync or ssh keys.
My new hosting service didn’t provide rsync or the ability to use SSH keys to upload changes to my website without entering a password every time.
I wanted to keep things fairly secure so no password could be stored in a file (it could be made public in a git repository), passed as a parameter to a script (it could be seen in a call to ps), or shown on the screen (it could be seen by another person).
I created two scripts. One calls Jekyll to compile the website and get just the files I need into a directory structure. The other is an expect script that manages the upload using scp.
Here is a line by line break-down of what the scripts do:
3. I use Oh-my-zsh which provides some nice out-of-the-box colouring and macros.
4. This method doesn’t delete files you don’t want anymore. You have to delete these manually at present.
5. Jekyll compiles your site.
6. Remove the staging directory that we will use to upload with scp.
7. Create the staging directory fresh.
8. Recursively copy the generated Jekyll site.
9. Delete all the files that are older than 2 weeks.
10. Call the Expect script to do the upload.
13. This is a script in the Expect language.
14. Hide the user output so that our password is not visible.
15. Calls a command that displays the password as saved previously in Keychain Access under the name xxxxx.
16. Matches anything that is returned that has at least one character.
17. Set the returned password to a variable whose name doesn’t sound like “password”.
18. Run SCP to connect to my website and copy the changed files. Note that you have to run a shell that can expand your * wildcard in the copy command. Expect can’t do it on its own.
19. Wait for the prompt to enter the password.
20. Send the password to the prompt.
21. Turn on user output.
22. Makes the output of the SCP command visible.
Feel free to add to the checks.
]]>https://www.codingame.com/home
https://www.codeeval.com/
https://www.hackerrank.com/
https://www.codewars.com/dashboard
Here are a few more of interest to me:
https://www.reddit.com/r/dailyprogrammer
http://up-for-grabs.net/#/
https://www.topcoder.com/
tl;dr - Problem solved!
Thankfully that I had backed up on Time Machine the night before, I booted off the backup drive and went into the Disk Utility. I couldn’t repair the SSD because I couldn’t unmount it. The Unmount button didn’t work. The computer wouldn’t boot into safe mode.
I rebooted from a Recovery DVD (although I could have done it the boot from the Time Machine too)
What unmounted it was selecting the disk in Disk Utility and selecting Info to get the name of the disk - for me it was disk0s2.
Closed Disk Utility and select Utilities » Terminal
cd / cd dev umount -fv disk0s2 I could now run Repair Disk and Repair Permissions in Disk Utility.
I had no problems with the Repair Disk (phew!) but there were permissions that were unable to be repaired by Disk Utility and the computer still kernel panicked when I booted.
Rebooted with Time Machine to the Terminal, unmounted the hard disk as above, and tried to restore from Time Machine Backup.
The system failed to see the hard disk.
Erased the partition, and it gave me a formatter error.
Tried to restore again from the Time Machine for the hell of it.
It couldn’t erase the disk.
I took out the SSD, and loaded it in a USB slot with a cable, then booted from the Recovery DVD and erased and reformatted the SSD. Success.
(6 hours later)
It then restored successfully from the Time Machine backup.
Moral of the story? Have recent backups and a Recovery DVD around (but I think the SSD or its cable might be on its way out :)
update: problem recurred. Will try a new Sata cable as suggested in numerous places on the internet.
(several days later)
update: new cable did the trick.
The clue that it wasn’t the SSD drive was that it didn’t have any errors on it when I ran Repair Disk in the Disk Utility. Phew!
]]>If you have access to the files of your WordPress site, navigate to wp-content/themes/ and either delete or change the name of the directory that contains the theme you selected.
This will revert the WordPress site to the default theme and you will be able to start to work out what happened.
]]>