HarpTabs.com :  Harptabs.com The fastest message board... ever.
Have ideas for the website? Like the way something works? Don't like something? Let us know here. 
Downloading the harptabs.com
Posted by: emrextreme (88.244.202.---)
Date: November 28, 2010 10:28AM

What's the entire size of the harptabs.com? I want to download to use it offline.

Options: ReplyQuote
Re: Downloading the harptabs.com
Posted by: harmusman (---.hari.cable.virginmedia.com)
Date: November 28, 2010 06:31PM

I don't know much of this downloading bit for use away from the actual internet connection but with 14,000+/12,000+ members and song titles it's gotta be pretty big to be carrying around.

Options: ReplyQuote
Re: Downloading the harptabs.com
Posted by: djlactose (---.static.optonline.net)
Date: December 02, 2010 02:53PM

You can not download the website to use offline. First off the way the site is designed you will be downloading an enormous amount of duplicate data and the entire site is several gigs which will use all of the site's bandwidth up and cause it to be shutdown.


Options: ReplyQuote
Re: Downloading the harptabs.com
Posted by: gene (---.dsl.rcsntx.swbell.net)
Date: December 03, 2010 02:40AM

Nick,
Is there a way to set it up so that all tabs (with index) and tabs only can be downloaded? How much space/bandwidth would we be talking about, then?

Options: ReplyQuote
Re: Downloading the harptabs.com
Posted by: meanon (---.107-92.cust.bluewin.ch)
Date: December 03, 2010 05:54AM

I don't think this issue is going to go away. Harptabs contains a massive amount of valuable content and for a number of reasons, people will want to know they can access even without an internet connection. I started thinking about an offline version when I was on holiday in a hotel in Italy at the beginning of this year. The hotel had a wireless lan but it was incredibly expensive and then, unluckily, harptabs was unavailable because the site had exceeded its bandwidth quota and was blocked for a couple of days. However, I was able to find some cached tabs using Google's search engine for those rainy days. But it got me thinking about what would happen if Harptabs just suddenly disappeared without trace? Horrible. I didn't even then have copies of the tabs I had posted. In the meantime I have suffered a little more nervousness when Google declared Harptabs to be an 'attack site'.
To cut a long story short, I decided to build an offline version for my own personal use. Since I earn my money in part programming in an arcane Linux scripting language called perl, it was not big challenge for me (much more challenging would be to earn my living playing the harmonica!). My first attempt simply letting a web crawler loose on the site failed because it kept going round in circles collecting the same file (but with a subtly different name) over and over again. So I downloaded a full index of the site and parsed out each separate link to a tab (almost 12,000) and fed these to the web crawler so it could collect only the tabs. It collected one HTML file for each tab but the HTML is a bit 'fat' with all the adds etc. so I parsed out the essential details leaving a skeleton HTML file for each Tab, then finally built a couple of indexes on it all. The data volumes involved: about 300 MB download (then just under 12,000 files uncompressed and with ads etc. ) and it took over 4 hours to collect. Compressed with zip and with 'lightweight' HTML it is about 16 MB and there is scope for further reduction. This does not include the forum, midi and other files etc. which I made no attempt to collect.
Don't ask me for a copy of it. As I said it is for my own personal use. Ideal would be if the webmaster built an downloadable version including his own choice of content / presentation (maybe with a Paypal link?) and included it on the site for download.


Options: ReplyQuote
Re: Downloading the harptabs.com
Posted by: djlactose (---.static.optonline.net)
Date: December 03, 2010 05:33PM

There are lots of issues with downloading the site as meanon said it is intentionally designed to cross link itself multiple times (it is a trick for search engine optimization). The other issue is if everyone started downloading the site we would run out of bandwidth, especially since the content changes so often.

Another issue is in the past we had discussed doing a book of all the tabs and selling that as another source of revenue for the site (the more money it makes the more time I can afford to spend on it). The issue is people had problems with their tabs being available any where other then on the website. They didn't even want them to be downloaded.

I have several things in the works for the site which may remove the need for what you are looking for. I am working on a total redesign from scratch of the entire site. This should require less bandwidth and provide additional security to everything. It will also give me flexibility to add many of the features people have requested over the years which just were not possible with the current code.

I am also working on APIs which would allow for content to be used in programs and on other websites (Harpsoft currently uses a version of this). I then plan on writing apps for several different devices which will be able to download and store locally tabs.

Right now my issue is having the time to do it. Many people know that I used to own my own consulting company but with the rough economy my clients went out of business which put me out of business. I am now have a full-time job which requires me to be on call 24 hours a day but I am at least afford free time now to do projects I want to do.


Options: ReplyQuote
Re: Downloading the harptabs.com
Posted by: meanon (---.107-92.cust.bluewin.ch)
Date: December 03, 2010 07:11PM

I'm looking forward to seeing some of these improvements. The content is the site's greatest asset and simple presentation with good searching / indexing / access counts for a lot. Times are clearly hard for the Webmaster at the moment so I've sent USD 25 by paypal. It won't make anyone rich (it wont even buy a reasonable harp) but it might just contribute to one year's registration fee for the internet domain.


Options: ReplyQuote
Re: Downloading the harptabs.com
Posted by: djlactose (---.static.optonline.net)
Date: December 03, 2010 07:37PM

Thank you for the donation. I have been trying to organize the new design to try and plan it out ahead of time, but as soon as I start having stuff available I will be posting it so people can start testing and playing with it. I have some really fun ideas which I think people will really like but I generally don't like to get people's hopes up until I am sure that I will be able to do it.


Options: ReplyQuote


Sorry, only registered users may post in this forum.
This forum powered by Phorum.