How to backup and sync your rooted Android phone automatically

1 03 2012

I have for a while now been working on a custom automatic sync solution for my HTC Hero running Froyo (Froydvillain ROM). As I am a Linux junkie and love scripts and hacks I wanted to do it all via cunning hacks and I’ve finally got it nailed.

This solution uses Scripting Layer 4 Android (SL4A) and Tasker alongside a custom ROM with rsync (any ROM should do so long as it has rsync). For those that don’t know, rsync is an awesome application that allows for remote backup and sync across machines. It turns out you don’t even need a ROM with rsync built in, as you can install an app that provides rsync, the app is called rsync backup for android and can be found here: https://market.android.com/details?id=eu.kowalczuk.rsync4android&hl=en

The only issue is you can’t call rsync from the command line using simply “rsync” since it isn’t in your systems path. However, if you use the following string instead,replacing calls to “rsync” with the following, the scripts still work: /data/data/eu.kowalczuk.rsync4android/files/rsync

SL4A is used to set out what to do via a script. You can write scripts in various languages in SL4A but I am using Bash as I am familiar with it. Rsync is used to actually handle the sync / backup and Tasker is used to launch the scripts when certain conditions are met.

I have created two scripts in SL4A, one backups my photos folder to my main photo folder on my server. The server runs the rsync daemon which rsync on the phone connects to. The other script does the reverse and copies a remote folder in my server that contains a bunch of music to my phone.

Tasker is set up with a profile that activates when my phone is plugged in and it’s between midnight and 7.00 am. This then connects to my WiFi network and then runs the two scripts via the SL4A plugin. Since I charge my phone each night this is effectively automatic.

The key here is getting permissions correct with rsync during the file transfer, as the memory card uses fat32 it hasn’t got any permissions. The rsync daemon doesn’t like this and errors out, hence the need for various settings. The second key here is exporting your password as an environmental variable. This is inherently insecure but since my server has multiple redundant backups and is only locally accessible I don’t care much. I could use trusted keys but I’m too lazy.

Here are the two scripts. First the music script that syncs from server to phone:

#rsync sync
export RSYNC_PASSWORD=password
DATE=$(date)
LOG=/mnt/sdcard/rsyncmusic.txt
echo rsync started $DATE > $LOG
TRY=1
rsync_com ()
{
DATE=$(date)
if [ $TRY = 15 ]; then
echo rsync failed, quitting on $DATE >> $LOG
exit
fi
sleep 10
echo rsync attempt $TRY started $DATE >> $LOG
rsync --progress -vHrltD --chmod=Du+rwx,go-rwx,Fu+rw,go-rw --no-perms --stats --password-file=/mnt/sdcard/scrt prupert@prupert::amusic /mnt/sdcard/amusic >> $LOG 2>&1
EXIT=$?
TRY=`expr $TRY + 1`
echo exit code is $EXIT >> $LOG
echo "********************" >> $LOG
}
rsync_com
while [ $EXIT != 0 ]; do
rsync_com
done
echo rsync finished $DATE >> $LOG
exit

The second script syncs the phones photos folder to my server:

#rsync sync photo
export RSYNC_PASSWORD=password
DATE=$(date)
LOG=/mnt/sdcard/rsyncphoto.txt
echo rsyncphoto started $DATE > $LOG
TRY=1
rsync_com ()
{
DATE=$(date)
if [ $TRY = 15 ]; then
echo rsync failed, quitting on $DATE >> $LOG
exit
fi
sleep 10
echo rsync attempt $TRY started $DATE >> $LOG
rsync -vHrltD --chmod=Du+rwx,go-rwx,Fu+rw,go-rw --no-perms --stats --password-file /mnt/sdcard/scrt /mnt/sdcard/DCIM prupert@prupert::apics >> $LOG 2>&1
EXIT=$?
TRY=`expr $TRY + 1`
echo exit code is $EXIT >> $LOG
echo "********************" >> $LOG
}
rsync_com
while [ $EXIT != 0 ]; do
rsync_com
done
echo rsync finished $DATE >> $LOG
exit

I have put some logging in to check progress and also some retry code that retries the sync if it timesout. It seems my HTC Hero's WiFi connection claps out after a while so the script retries up to 15 times to run successfully based on the rsync exit code.





get_iplayer startup script

19 02 2012

For funsies I am looking to create a web pvr, using get_iplayer, sickbeard and sabnzbd+ (though I am planning to stay on the legal side and only download what I can watch for free anyway).

Since there is no startup script for the get_iplayer web pvr program, I created one. You can find it here:

https://github.com/pruperting/code_snippets/blob/master/get_iplayer





XBMC Ubuntu Minimal Maverick Install Script

1 01 2011

As a little New Year’s present, I have updated a script that has been knocking around to allow you to install XBMC on a minimal version of Ubuntu Maverick.

If you simply install Ubuntu from the mini.iso containing the basic Ubuntu Maverick install, then on reboot, download my script using:

wget http://www.prupert.co.uk/scripts/xbmcmaverickrevo.sh

make it executable, using:

chmod a+x xbmcmaverickrevo.sh

then finally run it using:

sudo ./xbmcmaverickrevo.sh

I find it is best to run it again, once the script reboots the machine, as this fixes a few issues.

It is still a work in progress, as you can't yet shutdown / restart etc via XBMC since it seems the method to add permissions in Ubuntu 10.10 have changed and they conflict with the setup here, so I am still looking into that. To shutdown / reboot at the moment, I log in via SSH and issue:

sudo shutdown -hP now

Also, it is designed to work just with the Acer Aspire Revo 3610, but there is no reason why it wont work with any other device (apart from maybe the audio configurations).

Enjoy.

Download It Here





FINALLY: A Bash Progress Indicator for ffmpeg that WORKS

11 05 2010

I decided to write a progress indicator for ffmpeg video conversions. This was much much much harder than I ever anticipated. I was in for a tough time. Annoyingly, for what ever reason, ffmpeg doesn’t use Carriage Returns and Line Feeds in the expected way when it outputs its output during a conversion, resulting in two days of stress for a noobie bashophile like me.

Eventually, after hours of googling, I came across the solution. This script should be a very accurate indicator of the progress of a ffmpeg video conversion. It currently only tells you how far the conversion has progressed as a percentage, but I will soon add an ETA and time to it as well (now added, see update below). It does this, by looking at the number of frames your input video file has, and then reading what frame ffmpeg is currently on every ten seconds, and printing out the percentage.

You have to put in your own ffmpeg command if you want, mine is just for converting .mpg DVB-T recordings from MythTV to mkv files.
To use the script, call the script with the full path of the file to be converted afterwards, e.g.: “./ffmpeg-progress.sh /home/prupert/FIVE.mpg” and it gives you something like this:

UPDATE:

I have now added an ETA function, which, as far as I can tell, should be pretty accurate, I’ll be testing it tomorrow to see how well it works. It is based on the remaining number of frames to process and the current fps that ffmpeg is working at. I also plan to convert the script to python, if possible, to make it cross platform and give it a GUI. It will also be called YAFF (Yet Another Ffmpeg Frontend).

Essentially, the script turns ffmpeg output from this:

frame=42992 fps= 21 q=29.0 size= 217692kB time=1717.72 bitrate=1038.2kbits/s dup=12 drop=0
frame=43002 fps= 21 q=29.0 size= 217692kB time=1718.12 bitrate=1038.0kbits/s dup=12 drop=0
frame=43018 fps= 21 q=29.0 size= 217756kB time=1718.76 bitrate=1037.9kbits/s dup=12 drop=0
frame=43036 fps= 21 q=29.0 size= 217788kB time=1719.48 bitrate=1037.6kbits/s dup=12 drop=0
frame=43050 fps= 21 q=29.0 size= 217820kB time=1720.04 bitrate=1037.4kbits/s dup=12 drop=0
frame=43062 fps= 21 q=29.0 size= 217852kB time=1720.52 bitrate=1037.3kbits/s dup=12 drop=0
frame=43078 fps= 21 q=29.0 size= 217884kB time=1721.16 bitrate=1037.0kbits/s dup=12 drop=0
frame=43090 fps= 21 q=29.0 size= 217916kB time=1721.64 bitrate=1036.9kbits/s dup=12 drop=0
frame=43106 fps= 21 q=29.0 size= 217948kB time=1722.28 bitrate=1036.7kbits/s dup=12 drop=0
Last message repeated 2 times
[mpeg2video @ 0x8f7ebf0]ac-tex damaged at 15 0
[mpeg2video @ 0x8f7ebf0]concealing 90 DC, 90 AC, 90 MV errors
frame=43122 fps= 21 q=29.0 size= 217981kB time=1722.92 bitrate=1036.4kbits/s dup=12 drop=0
frame=43140 fps= 21 q=29.0 size= 218045kB time=1723.64 bitrate=1036.3kbits/s dup=12 drop=0
frame=43155 fps= 21 q=29.0 size= 218077kB time=1724.24 bitrate=1036.1kbits/s dup=12 drop=0
frame=43168 fps= 21 q=29.0 size= 218141kB time=1724.76 bitrate=1036.1kbits/s dup=12 drop=0
frame=43182 fps= 21 q=29.0 size= 218205kB time=1725.32 bitrate=1036.1kbits/s dup=12 drop=0
frame=43194 fps= 21 q=29.0 size= 218237kB time=1725.80 bitrate=1035.9kbits/s dup=12 drop=0
frame=43207 fps= 21 q=29.0 size= 218269kB time=1726.32 bitrate=1035.8kbits/s dup=12 drop=0

In to this:

/home/rupert/Downloads/Village of the Damned.mpg..mpg has 172447 frames, now converting
ffmpeg PID = 24868
ffmpeg: 43062 of 172447 frames, progress: 24% and ETA: 1h 42m 41s

Which must be a good thing in anyone's book ;)

The script is hosted at github, enjoy:

Creative Commons License
YAFF by Rupert Plumridge is licensed under a Creative Commons Attribution-Non-Commercial-Share Alike 2.0 UK: England & Wales License





Python Android Weather Forecast Script

28 04 2010

I am a proud owner of an HTC Hero Android phone.  I’ve recently  been messing around with one very cool looking application, that allows you to run your own scripts on the phone. This app is called Android Scripting Environment (ASE) and it is truly ace. You can use it to write and run your own Python, Shell (Bash), Lua and Ruby scripts directly on your phone. It also hooks into various Android features, allowing you to make use of various cool Android functions, one of which is getting your location and doing cool things with that information.

I’ve been wanting to get into Python for a long time and I decided that now was a good a time as any. ASE comes with a few sample scripts to get you going, one of which gets the current weather conditions for your current location and speaks it outloud (say_weather.py which calls weather.py). As cool as this is, being told the current conditions isn’t that useful, I wanted to know the forecast for the next day. I thus delved in and did lots of Googling, mainly using this awesome page that teaches you how to parse XML files (which is the format Google delivers its weather forecast info in). The biggest issue was the XML feed from Google doesn’t clarify the difference between the five day forecasts until you get to an actual data reading. The XML feed looks like this:

<xml_api_reply version="1">
      <weather module_id="0" tab_id="0" blah>
		<forecast_information>
			<city data="Bristol, Avon"/>
                        <postal_code data="bs167eb"/>
                        <latitude_e6 data=""/>
                        <longitude_e6 data=""/>
			<forecast_date data="2010-04-28"/>
                        <current_date_time data="2010-04-28 12:50:00 +0000"/>
                        <unit_system data="US"/>
		</forecast_information>
		<current_conditions>
			<condition data="Clear"/>
			<temp_f data="64"/><temp_c data="18"/>
			<humidity data="Humidity: 52%"/>
			<icon data="/ig/images/weather/sunny.gif"/>
			<wind_condition data="Wind: S at 16 mph"/>
		</current_conditions>
		<forecast_conditions>
			<day_of_week data="Wed"/>
			<low data="51"/>
			<high data="66"/>
			<icon data="/ig/images/weather/chance_of_rain.gif"/>
			<condition data="Chance of Rain"/>
		</forecast_conditions>
		<forecast_conditions>
			<day_of_week data="Thu"/>
			<low data="46"/>
			<high data="60"/>
			<icon data="/ig/images/weather/chance_of_rain.gif"/>
			<condition data="Chance of Rain"/>
		</forecast_conditions>
		<forecast_conditions>
			<day_of_week data="Fri"/>
			<low data="48"/>
			<high data="55"/>
			<icon data="/ig/images/weather/chance_of_rain.gif"/>
			<condition data="Chance of Rain"/>
		</forecast_conditions>
		<forecast_conditions>
			<day_of_week data="Sat"/>
			<low data="44"/>
			<high data="59"/>
			<icon data="/ig/images/weather/chance_of_rain.gif"/>
			<condition data="Chance of Rain"/>
		</forecast_conditions>
	</weather>

Now, I know nothing about XML at all, but I found it impossible to get the first actual forecast from the XML data, as each forecast was simply called "forecast_conditions", using the method in the original weather.py. Thus, after reading the handy tutorial from faqs.org (listed above) I realised I needed to use the nested childNodes function of the xml.dom function that you use in Python to parse XML files.

After lots of trial and error, I worked out the following. A node is a name given to a section named via

<some_name>

. First, you assign a name (in this came dom) to the XML you are reading by doing: dom = minidom.parseString(xml_response), then I want to get information out of the node. To move through the nodes, you use the following command dom1Node = dom.firstChild where dom was the name of the original XML file. All this does is move to the first node and give it a name (dom1Node in my case), thus we are now at

<xml_api_reply version="1">

. You then use dom2Node = dom1Node.firstChild to move to the next node and give it a name, thus we are now at

<weather module_id="0" tab_id="0" mobile_row="0" mobile_zipped="1" row="0" section="0">

. Finally, we have got somewhere, because the childNodes (i.e. the ones beneath the node we are at) are the ones with interesting info, such as

<forecast_information>

and most importantly

<forecast_conditions>

. So, finally, we can now assign names to these childNodes that we want. So we use current = dom2Node.childNodes[1] for the node (the numbering starts at 0 not 1) and forecast = dom2Node.childNodes[2] for the first forecast. Finally, to get the actual data out of those nodes, we use the following command data['flow'] = forecast.getElementsByTagName('low')[0].getAttribute('data'). What this does is give the name "flow" to the data element with the Tag name "low" from the "forecast" node that we defined just above. So, essentially, it navigates to and extracts the value "51" from it.

Thus, using this technique, you can get all the forecast data you want. I therefore ended up with this script, forecast.py, to replace the weather.py, to return forecast data:
And I then edited the say_weather.py to say the forecast as well as the current conditions, resulting in say_forecast.py:

The only problem I had with this was that the Google XML feed only appears to give the temp in fahrenheit, not celcius. I can see that you can set the iGoogle to display the temp in celcius, but I couldn't see how to get that info from a feed.

Now, the next thing on the cards is to make a widget that displays this data on the homescreen. I was thinking of saving the output of the python script to a text file and then make a widget that reads that text file and displays the data. If I can do that, I can then modify the python script to do all sorts of cool things (get news, info from my mythtv HTPC etc).

Sadly, it seems on my version of Android (Cupcake (1.5)) you can't call ASE scripts automatically via Locale or Tasker, since I get an ASE Force Closed message whenever I try to :( So these currently have to be run manually. Still, this is my first piece of Python coding and it was much easier than I thought it would be, once I got my head round it all.





Simple Bash Progress Script

21 04 2010

So, I wanted a way to monitor the progress of a Bash script I was running, since it could take a long time to run on occassion and I wasn’t sure if it had failed or not.

After some Ubuntu Forum googling, I came up with this:

#!/bin/bash
time=( $(tac /var/log/rsback/work-daily | grep -m 1 "total runtime" | awk '{ print $6 }') )
echo "last running time was" $time
/bin/rsback -v work-daily >> /var/log/rsback/work-daily &
pid=$!
trap "
#[ -e /proc/$pid ] && kill $pid
#" EXIT


while [ -e /proc/$pid ]; do
sleep 10
runtime=( $(ps ax | grep -m 1 zenity | awk '{print $4}') )
echo "#" $runtime
done | zenity --width 500 --progress --pulsate --title="work-daily running time: $time" --text="running time:" --auto-close
retval=$?


# If the progress bar was canceled, give a warning.
if [ $retval -ne 0 ]; then
zenity --title="work-daily rsback" --error --text="the work-daily rsback was cancelled" || exit 1
kill $pid
fi
time=( $(tac /var/log/rsback/work-daily | grep -m 1 "total runtime" | awk '{ print $6 }') )
zenity --info --title "Work rsback" --text "work rsback done in $time"
exit

It seems to work pretty well, though obviously this is customised for my particular needs (so it uses the command's own log location etc). I thought I would share it with the world, since it took me a while to figure out. When I get the chance, I'll update it to it can work with any command or script ;)





Automatically Update FFMPEG and x264 From Source / SVN In Ubuntu

16 04 2010

Whenever I wish to install and use the rather excellent video and audio transcode programs ffmpeg and x264 I have used FakeOutdoorsman’s excellent How To from the Ubuntu Forums. However, constantly typing out all the commands, especially when updating, is a pain in the arse.

I thus decided, with FakeOutdoorsman’s permission, to write some scripts to automate the task. They are, I hope, fairly comprehensive and allow a user to build and install the latest version of ffmpeg and x264, from source for Ubuntu Hardy, Intrepid, Jaunty, Karmic and Lucid. They allow this to be done automatically, after a few questions have been answered.

There are four scripts in total, one for the initial install, one for keeping everything up-to-date, one to reverse the changes with minimal impact and one to completely reverse all changes (this one I recommend no-one using, hence it is depreciated, but it is there in case some one wants it).

I am certainly not the best at bash, but I have done lots of Googling and I think the script is fairly robust, though I am sure there are errors.

I will be adding to the script over time, adding new features, combining things, adding better error checking etc, so for now consider them BETA releases. They have been tested on Lucid and Karmic only, but they appeared to work.

Please submit any questions here or at the project hosting page.

So, where are the files?

You can get all the scripts here:

http://code.google.com/p/x264-ffmpeg-up-to-date/

Have fun!





MythTV Recordings: Rename, Move and Delete From The Database

12 11 2009

So, I have been using MythTV for a while now and it is a.w.e.s.o.m.e.. However, I had one problem, since I transcode and store all my videos on my Ubuntu server and not my MythTV box (which is off most of the time via mythwelcome) I had to manually transfer recorded files to my Ubuntu server via mythweb.

I thought this was a bit of a hokey solution, so I asked around if there was a script that allowed you to rename MythTV recordings (to a more human readable version), move them to another location and remove them from the MythTV database. There were some solutions out there, but they all involved the recorded files being kept in the MythTV database.

mythtv

So, I wrote my own script. This script renames the MythTV files to a nice human understandable version using mythrename.pl, it then moves the recordings to a folder of your choice (in my case a mounted samba share on my server), it then uses myth.find_orphans.pl to delete the moved recordings from the mythTV database (I don’t know if it will still remember that these files were previously recorded so it doesn’t rerecord them or not, I guess time will tell).

So, without further ado, here is the script, in lovely Bash (the script is called mythtvremodel (mythtv REname MOve DELete, geddit?):


#! /bin/sh
### BEGIN INIT INFO
# Provides: mythtvremodel
# Required-Start:    $local_fs $syslog $remote_fs dbus
# Required-Stop:     $local_fs $syslog $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: rename, move delete recordings
### END INIT INFO
# script to rename files recorded from mythtv and then to move them and delete them
#made by prupert.co.uk
#creative commons licence, whichever is the most FREE (as in beer)


#lets put in some variables so others can use this
#location of log file, you can choose anything and any name
log=/path/to/mythtvremodel.log
#this is the full path to mythrename.pl
mrename=/location/to/mythrename.pl
#this is the full path to your recordings folder
record=/var/www/mythweb/data/recordings/
#this is the extension your recordings are stored as, usually mpg or nuv
ext=mpg
#this is the full path to where you want your files moved to
dest=/path/to/folder/to/move/recordings/to/
#this is the full path to myth.find_orphans.pl
morphan=/path/to/myth.find_orphans.pl
#mythtv database password for myth.find_orphans.pl
pass=yourmythtvdatabasepassword


# first, run mythtvrename to get nice names for the recordings
echo starting mythtvremodel on `date "+%m/%d/%y %l:%M:%S %p"` > $log
perl $mrename --underscores --format %T-%S-%d%m%y >> $log
# use find to search for all recorded files in the recording folder and move them to the TV folder
find "$record" -name "*.$ext" >> $log
find "$record" -name "*.$ext" -exec mv {} "$dest" ; >> $log
# use mythorphan to remove the now missing recorded files from the database
perl $morphan --pass=$pass --dodbdelete >> $log
#that should be it
echo stopping mythtvremodel on `date "+%m/%d/%y %l:%M:%S %p"` >> $log
exit 0

And there you have it. Now, since I run mythbuntu, I have put this script in /etc/rc0.d (thats rc zero . d) and have named it K03mythtvremodel, thus it is run when my system shuts down, but before all the samba shares and drives are unmounted. As I said, since I use mythwelcome to shutdown my machine when idle, this seemed like the best solution.

Hope it helps someone!





MediaPortal Comskip and Transcode Bat and Bash Scripts

8 09 2009

As I mentioned earlier, I use MediaPortal to record DVB-T broadcasts. As these end up quite big, I then use two scripts to check the recordings for adverts and to transcode the recordings to a mkv file (smaller and better supported).

This solution isn’t very elegant, since it is for my personal setup which is a little unusual, in that the recordings are done via a Windows XP MediaCentre PC, they are then analysed for adverts using Comskip and are then transferred to my Ubuntu-based server, which then transcodes the files to an MKV file and the adds Chapter markers to those MKV files using the file from Comskip. Thus there script is in two parts, one a Windows Bat script and the other is a Ubuntu Bash script.

The scripts are below. First, the Windows Bat script that runs the recorded file through Comskip and then copies it over to the server. Since I wanted to only run one script for all the recordings made in an evening, the script takes the file name from MediaPortal, which passes it to the script after the file is finished recording, using the Comskip plugin. This explains all the weird “Echo” in front of all the terms, since basically it is putting all the commands into another script, which is then run early each morning.

BASH

BASH

@ECHO OFF
ECHO ECHO.
ECHO Rem Batch commands for %1 START >> "C:scriptscomskiptrans.bat"
ECHO CD "C:scriptscomskip" >> "C:scriptscomskiptrans.bat"
ECHO START "" "C:Program FilesVideoLANVLCvlc.exe" >> "C:scriptscomskiptrans.bat"
ECHO %%SystemRoot%%explorer.exe "Z:" >> "C:scriptscomskiptrans.bat"
ECHO ECHO ************************** ^>^> "%~dpn1.PostLog.txt" >> "C:scriptscomskiptrans.bat"
ECHO ECHO Post Processing started on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:scriptscomskiptrans.bat"
REM ECHO ECHO Starting mpeg2repair %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:scriptscomskiptrans.bat"
REM ECHO mpeg2repairHelper.exe %1 %1-fixed.ts >> "C:scriptscomskiptrans.bat"
REM ECHO ECHO mpeg2repair finished on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:scriptscomskiptrans.bat"
ECHO ECHO Comskip started on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:scriptscomskiptrans.bat"
ECHO comskip %1 >>"%~dpn1.PostLog.txt" >> "C:scriptscomskiptrans.bat"
ECHO ECHO Comskip finished on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:scriptscomskiptrans.bat"
ECHO ECHO %%date%% %%time%% ^>^> "Z:new_files.log" >> "C:scriptscomskiptrans.bat"
REM ECHO ECHO ComClean started on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:scriptscomskiptrans.bat"
REM ECHO mencoder "%~dpn1.ts" -edl "%~dpn1.edl" -alang eng,nar -noskip -vf harddup -oac copy -ovc copy -of mpeg -o "%~dpn1_clean.ts" >> "C:scriptscomskiptrans.bat"
REM ECHO ECHO ComClean finished on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:scriptscomskiptrans.bat"
ECHO ECHO New file %1 being moved to Server ^>^> "Z:new_files.log" >> "C:scriptscomskiptrans.bat"
ECHO ECHO Tidying started on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:scriptscomskiptrans.bat"
ECHO %%SystemRoot%%explorer.exe "Z:" >> "C:scriptscomskiptrans.bat"
ECHO CD "%~dp1" >> "C:scriptscomskiptrans.bat"
REM ECHO ren *.edl *.ts.edl >> "C:scriptscomskiptrans.bat"
REM ECHO ren *.txt *.ts.txt >> "C:scriptscomskiptrans.bat"
ECHO ren *.chap *.ts.chap >> "C:scriptscomskiptrans.bat"
ECHO ECHO Moving Clean TS to Server directory on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:scriptscomskiptrans.bat"
REM ECHO IF EXIST "%~dpn1_clean.ts" del %1 >> "C:scriptscomskiptrans.bat"
ECHO xcopy %1 Z: >> "C:scriptscomskiptrans.bat"
ECHO IF EXIST "Z:%~n1.ts" del %1 >> "C:scriptscomskiptrans.bat"
REM ECHO xcopy "%~dpn1.edl" Z: >> "C:scriptscomskiptrans.bat"
ECHO del "%~dpn1.txt" >> "C:scriptscomskiptrans.bat"
ECHO xcopy "%~dpn1.ts.chap" Z: >> "C:scriptscomskiptrans.bat"
ECHO ECHO TS and others moved to Server directory on %%date%% %%time%% ^>^> "%~dpn1.PostLog.txt" >> "C:scriptscomskiptrans.bat"
ECHO ECHO %~n1.ts moved to Server on %%date%% %%time%% ^>^> "Z:new_files.log" >> "C:scriptscomskiptrans.bat"
ECHO TASKKILL /IM vlc.exe >> "C:scriptscomskiptrans.bat"
ECHO Rem Batch commands for %1 FINISH >> "C:scriptscomskiptrans.bat"

Thus this script creates the trans.bat file, which is then run via a scheduled task. All the stuff starting with REM are lines that the script no longer uses, but that contain interesting code that might be useful in the future. This includes using a program called mpeg2repair, that was meant to clean the recording in-case there were any errors during transmission, but the program error-ed out too many times. Another line was also meant to use Mencoder to cut the adverts out directly using an EDL file created by comskip, but this resulted too often in audio/video sync problems, I think as a result of the original .ts file having some errors in it, so I REMed out that line as well.

The second script is my Ubuntu bash script, which is run every morning, after the trans.bat script (created by the script above) is run.

#!/bin/bash
# Written by FakeOutdoorsman and updated by mysoogal further updated by prupert
# Attribution-Noncommercial 3.0 Unported
# http://creativecommons.org/licenses/by-nc/3.0/deed.en
# trackback http://ubuntuforums.org/showthread.php?t=960627
echo trans.sh started on `date "+%m/%d/%y %l:%M:%S %p"` > /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.ts" >> /mnt/media/documents/ruperts/TV/trans.log
chmod -R 777 /mnt/media/documents/ruperts/TV
find /mnt/media/documents/ruperts/TV -name "*.ts" -exec /home/server/Desktop/handbrake/HandBrakeCLI -i {} -t 1 -c 1 -o {}.mkv -f mkv -e xvid -b 1200 -2  -a 1 -E faac -B 160 -R 0 -6 dpl2 -D 1 -C 2  -v '{}' ; >> /mnt/media/documents/ruperts/TV/trans.log
echo trans.sh finished on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
echo attempting to add advert chapter markers on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.mkv" -exec /home/server/scripts/mkvmerge.sh "{}" ; >> /mnt/media/documents/ruperts/TV/trans.log
echo advert chapter markers added on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
echo starting clear up on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
cd "/mnt/media/documents/ruperts/TV/" || { echo >&2 "Source folder not found"; exit 1; } >> /mnt/media/documents/ruperts/TV/trans.log
find . -type f -name "*.ts" -exec sh -c '[ -f "${1%.ts}.ts.mkv" ]' _ {} ; -delete >> /mnt/media/documents/ruperts/TV/trans.log
find . -type f -name "*.ts" -exec sh -c '[ -f "${1%.ts}.mkv" ]' _ {} ; -delete >> /mnt/media/documents/ruperts/TV/trans.log
find . -type f -name "*.ts.mkv" -exec sh -c '[ -f "${1%.ts.mkv}.mkv" ]' _ {} ; -delete >> /mnt/media/documents/ruperts/TV/trans.log
find . -type f -name "*.ts.chap" -exec sh -c '[ -f "${1%.ts.chap}.mkv" ]' _ {} ; -delete >> /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.mkv" -exec mv {} /media/video/Video ; >> /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.edl" -exec rm {} ; >> /mnt/media/documents/ruperts/TV/trans.log
find /mnt/media/documents/ruperts/TV -name "*.txt" -exec rm {} ; >> /mnt/media/documents/ruperts/TV/trans.log
echo ts files deleted and moved script stopped on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log

This is run by a cron job every morning. It uses Handbrake to transcode the .ts file to a .mkv file. It then runs a third script (below called mkvmerge.sh) to use mkvmerge to add chapter markers to the mkv file that Handbrake created. Finally, it moves the mkv file to where all my Videos are stored. At the very end, there are some commands to delete the .ts files and the intermediate .mkv files ONLY if the mkv file has been correctly created.

Here is the mkvmerge.sh script:

#!/bin/bash
echo mkvmerge.sh started on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/trans.log
VIDEO=$1
ENC=${VIDEO##*/}
mkvmerge --chapters "${VIDEO%/*}/${ENC%%.*}.ts.chap" "$VIDEO" -o "/mnt/media/documents/ruperts/TV/${ENC%%.*}.mkv"

This script is called by the main Ubuntu trans.sh script above.

I also have a separate script that I use to transcode the .m2ts files produced by the Sony PS3's PlayTV program:

#!/bin/bash
echo transm2ts.sh started on `date "+%m/%d/%y %l:%M:%S %p"` > /mnt/media/documents/ruperts/TV/transm2ts.log
find /mnt/media/documents/ruperts/TV -name "*.m2ts" >> /mnt/media/documents/ruperts/TV/transm2ts.log
chmod -R 777 /mnt/media/documents/ruperts/TV
find /mnt/media/documents/ruperts/TV -name "*.m2ts" -exec /home/server/Desktop/handbrake/HandBrakeCLI -i {} -t 1 -c 1 -o {}.mkv -f mkv -w 720 -e xvid -b 1200 -2 -a 1 -E faac -B 160 -R 0 -6 dpl2 -D 1 -C 2 -v '{}' ; >> /mnt/media/documents/ruperts/TV/transm2ts.log
echo transm2ts.sh finished on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/transm2ts.log
echo starting clear up on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/transm2ts.log
cd "/mnt/media/documents/ruperts/TV/" || { echo >&2 "Source folder not found"; exit 1; } >> /mnt/media/documents/ruperts/TV/transm2ts.log
find . -type f -name "*.m2ts" -exec sh -c '[ -f "${1%.m2ts}.m2ts.mkv" ]' _ {} ; -delete >> /mnt/media/documents/ruperts/TV/transm2ts.log
find . -type f -name "*.m2ts" -exec sh -c '[ -f "${1%.m2ts}.mkv" ]' _ {} ; -delete >> /mnt/media/documents/ruperts/TV/transm2ts.log
find /mnt/media/documents/ruperts/TV -name "*.mkv" -exec mv {} /media/video/Video/TV ; >> /mnt/media/documents/ruperts/TV/transm2ts.log
echo m2ts files deleted and moved script stopped on `date "+%m/%d/%y %l:%M:%S %p"` >> /mnt/media/documents/ruperts/TV/transm2ts.log
exit

Because Comskip can't be run on .m2ts files, I don't call the mkvmerge.sh script, since there is no .chap file to use.

I would freely admit that these scripts aren't very user friendly, since they are customised just for my setup. However, if readers would prefer a useful version that was more user-friendly, I could create that. It is possible to convert the Ubuntu Bash script into a Bat script for Windows. However, I don't think there is any alternative to the mkvmerge file for Windows that the Ubuntu script uses.

Also, sadly, the devs of Handbrake are a little blinkered. They created the CLI interface so you can run Handbrake via the commandline, which is great. The CLI also creates Exit codes, which is very useful for scripting. However, the Exit codes only tell you if the CLI worked correclty, not if Handbrake worked correctly. So if the transcode failes, but the CLI worked corretly, it will exit as if nothing went wrong. Sadly the Handbrake devs don't see this as a problem. Kind stupid imho, but hey, it's their loss. Thus, there is no way of telling if the transcode worked correctly or not, the only way to check is to see if the file you want to be created is there...doh!





Bash and Windows batch script file name variable conversion

15 07 2009

I’ve been getting into Bash and Windows Batch scripting recently (as part of my ongoing battle to get my HTPC working correctly with MediaPortal).

One problem I kept on coming across was how to pass different variations of the variable you pass to your script, sometimes I wanted the full path to the file, other times I just wanted the filename with the file extension, other times I wanted just the folder the file was located in.

It took a fair bit of googling and experimentation with terms to find the right information, so I have decided to amass it all here to help others (and me).

For Bash Scripting

(taken from linuxgazette)

    Given:

      foo=/tmp/my.dir/filename.tar.gz
    We can use these expressions:
    path = ${foo%/*}
    To get: /tmp/my.dir (like dirname)
    file = ${foo##*/}
    To get: filename.tar.gz (like basename)
    base = ${file%%.*}
    To get: filename
    ext = ${file#*.}
    To get: tar.gz

For Windows Batch Scripting

(taken from imagemagick)

%~1 expands %1 removing any surrounding quotes (“)
%~f1 expands %1 to a fully qualified path name
%~d1 expands %1 to a drive letter only
%~p1 expands %1 to a path only
%~n1 expands %1 to a file name only
%~x1 expands %1 to a file extension only

These two pieces of info can thus be used to pass any variable or variation of that variable to a command in your script.

Using these tools I have been able to get MediaPortal to automatically scan each new recording for adverts using comskip and then transfer the recording, alongside the necessary advert info files to my network hard drive on my server using a batch script. Then the server runs a bash script each morning to scan for new files on the new recordings file, convert them to MKV using Handbrake and add chapter markers and then move the MKV files to my networked videos file. Cool!








Follow

Get every new post delivered to your Inbox.