About HasRoot

The story of how I made a site I didn't plan to and how we got from one Java file to a one page site to here. Contains the history of the major feature changes and a summary of the entire history, obviously a lot of detail is omitted such as the massive options rework, streamer profiles, streamer activity graph pages, the redesigns, new menu bars or change of APIs.

The beginning

So how I go into GTA RP seems like a good place to start, when the first initial blowup happened when lirik started playing on SoE my RP experience was limited to playing DnD in highschool and then some various similar tabletop games with friends as I got older. Truth being told I wasn't really RPing back then, just playing games with friends and always found the RP aspect hard or cringy to do myself. My first experience with 'real' RP was with RollPlay, I was a big fan of SC2 when it came out so was a fan of State of the Game and followed them when they started doing DnD content. However when GTA RP blew up I was incredibly sceptical and thought it seemed weird as fuck but decided to check it out. I started watching Giant Waffle one day and found it hilarious. This lead to me getting into buddha, finkone, eli, ghillie guy, sarapocalypse, sirpinkelton, tastytv and more and before I knew it I was watching vods to make sure I wouldn't miss any of the ongoing stories. These vods were repeatedly muted and sometimes even mass purged if someone leaked the server password in OOC and at the time the VODs were very laggy for me. So for my own purposes I ended up making a Java application to record live streams using livestreamer.

At first I could tell it to record someone who is live and keep trying untill it is convinced they are offline or I could set a time window to check for a streamer and record if live, it was pretty crappy but it sorta worked initially. I started to get annoyed by these limitations and eventually decided to look for a Java twitch API and after finding a couple none of which I liked I decded to write my own. The app could now check when someone went online or offline and then I figured I may as well use it to see who is on when I am watching live too. More features were added for title/game changes and I then went looking for streamers who played on SoE via forums and other sites. I found that there was some efforts to maintain lists of the streamers but most of them were out of date so I updated my Java app to crawl the top 500 GTA streams using titles to flag anyone who may be on the server. It wasn't long after this that the SoE rift happened with a mass exodus of RPers which would eventually lead to the creation of FamilyRP. I with almost all of the viewers followed them to their new home and it became the focus for my app.

One day when looking around the forums I ended up finding The Family RP wiki which was was a great resource for character data. Until this time I had only been adding character information myself in my app and the data was very shoddy. I started pulling their data and working on a system to store it in a better way as it was still 100% based on an XML file and moved to using a database as the back end. With this done I started working on the Family RP wiki for awhile, generating the streamer table for them with the character and streamer data I had.

At this point I realised my app was becoming pretty useful for people but knew that a Java app would not be popular, I considered making a desktop version in a more user friendly language with a UI but that wasn't something I was very interested in. I eventually decided maybe I should just make a site, my app was already pushing the data to a database anyway so I could just have the website be a UI for it. My current hosting wouldn't allow me to run a Java app on the box and I wasn't about to go fork out money for a random project that no one may ever use. So I reached out to my friend Sam and asked if he would be crazy enough to let me run a Java app on his side businesses server and that it was a 'small site that may explode in popularity, but don't worry it will probably never happen and I will make sure it is scaleable just incase'. Weirdly he said yes.

The site was born

So I started work on the site and it had been awhile since I had worked on HTML/ CSS / PHP and I was never good at web design so the site was rough at the start. With scalability in mind I created a custom layer over phpfastcache and used smarty for page cacheing and got to work on writing the JavaScript layer to poll the back end for status updates. After awhile a one page Family RP stream list with characters was born. There was very little options if any, the colour scheme was literally shades of grey with a half intent for it to be a placeholder and half hoping someone might take pity and create a nicely designed stylesheet for it. So on Aug 30, 2017 I went and posted it on the Family RP forums and waited to see what would happen.

The launch

Initially traffic was low, like really low in 2017 we peaked at 22 unique visitors in a month. I had put a fair amount of effort into the site and my concern for scalability at that moment seemed overkill (though later id be thankful I spent so much time on it) and I was a little disappointed however I was using it myself and so did a handful of other people. When I was a bit disillusioned with it though LizardWizardAlien reached out to me on Reddit about how great the site was and how useful it was and gave me feedback and suggestions on how to improve it. It was him who suggested the colour scheme we still currently use and for awhile I would often pester him for feedback about new updates I was working on. Thanks again LizardWizardAlien, maybe one day I will get to custom sorted stream containers lol.

With the site mainly working I started looking into other RP servers for a variety of reasons, quickly focusing on No Pixel and SoE once again. The state of the servers at that time was nothing like today but adding more servers to the backend wouldn't take a monumental amount of work so I figured I may aswell. From the log files of the git repo it was 19/10/2017 when I commited a large number of changes to support these servers, branching out from just being a website for FamilyRP and becomming a website for GTA RP in general.

I found the thumbnails letting me get a glimpse of what people were up to at a glance very useful so I started to think about what else I could add that would be cool. I have been into development since I was a teenager and I remembered a blog about reconstructing massive datasets not from a database or object file but just by reprocessing massive log files. The Java app itself was basically outputting a massive log file for my own uses so I figured throw all of those state events into a database and they'll be useful one day. So without any real plan with what I was going to do with them that is what I did and called them channel snapshots.

For awhile I didn't do much with that data but I found myself checking it to see who was on the night before and roughly for how long, so I figured a web page to access those kinds of features would be useful. I wasn't sure what the best way would be to accomplish this, just list them in a table based on who was on last night? That plan didn't appeal to me but this is sort of what inspired the streamer activity pages. So I started looking around and found vis.js graphs and I thought maybe I could wrangle the timeline API into something suitable.

History page

This took awhile and wasn't as easy as I had hoped but eventually I had built a history page which listed the GTA RP histories of the tracked streamers. At this point there wasn't thumbnails and there wasn't hover able jump to time features, these would come later and require more data than I had available at the time. The data itself was kind of messy back then too as it was only using my own data and not any official data for durations of vods etc.
I thought it would be cool to be able to click to go to the VOD but that data wasn't something I had been tracking at all yet and it wasn't information that was easily gleaned from the API. The only solution I could find was to just check the streamers video list via the API and grab the VOD that has the recording status.

So I had to start tracking this data in the channel snapshots too and I believe I wiped any past held channel snapshots because of this and a bug that was causing duplication of snapshots.I also quickly realised when pulling data from the channel snapshots table that it was slow to pull a lot of data and figure out a greater context from the individual snapshots. This required me to pre process data and the online histories table was born which was a more summary based view of the snapshots. I had to actually generate online histories from the existing channel snapshots over a period of 24 hours in small batches to facilitate this. Also at this stage the data would get scuffed if there was major back end problems or if the backing app crashed (which it didn't often do it but it happened occasionally). There would be many iterations of how exactly this data was stored and generated to improve performance and reliability.

I always thought it was be insanely cool if you could hover to see the VOD thumbnails but this wasn't exposed by twitches official API in any way and I wasn't sure how feasible it would be. I started to experiment with reversing how twitches code worked for this while being a little worried I may one day get an email telling me not to do this anymore but figured it may not even be feasible to add well so what's the worst that could happen. I quickly realised I needed to pull more data than I had to smooth out the data.While I worked out how to display a thumbnail relative to a VOD timeline, I needed 100% accurate start time and VOD lengths to determine the right segment of the thumbnails to display. The thumbnails for twitch vods use a fixed number of thumbnails regardless of the length of the VOD so the time period each correspond to is variable.I didn't track that as until this point I had only been using the VOD id in my histories which was perhaps short sighted.

So I had to start tracking it and that also meant I had to pull data on past vods as well. I wrote some code that would batch pull these over days and I think it took nearly 30 hours to collect all the data while trying to be respectful of twitch rate limits and my usage on the server not encroaching on other paying users. When the data all came in, it worked, it actually worked and it worked surprisingly well. I was kinda shocked to be honest but thought now that is fucking cool. It started making me think of what else could be done with the history page.

I had always liked the idea of sharing 'views' of the history page and had allowed for that in earlier iterations but it was rarely used by people.It did however spark an idea, it would use a timestamp parameter and with all of the VOD history work I had been doing lately and generating VOD offsets from where you click in a VOD I started to think wouldn't it be cool if you could just go see the history from a clip?

Clips

So I had to start looking into clips, this was one area of the twitch API I had previously completely ignored. I quickly realised it wouldn't be that hard to locate the precise moment a clip corresponded to though some database changes were required. I implemented a way for the website to pull clips from the twitch API and generate the data. I did realise however that the data generation was slow as there was now a lot of data to search through for each clip. I also wasn't really happy about the PHP layer having to pull clip data from twitch, I had made a ReCAPATCHA thing for it but I figured users wouldn't like doing that all the time and it just seemed a bit of a half realised update / feature.

So I decided I could just batch process the clips somehow and store the data as clip lookups. By having the back end polling clips too this meant it solved another issue which was viewer counts not being updated and thus would be useless.This meant I had to write code to discover clips, I started with the GTA V top and new clips by day month etc but found that it was missing some that I would see on streams or on the subreddit. I then decided id bite the bullet and make it crawl clips from streamers who streamed any GTA RP within the last 48 hours, going through their channel clips by top and new. Now I had a lot of clip information, a lot and as you may have have noticed the trend by now, when I have data and think it can be useful I use it and thus the clips page was born. It was never my intention to make a clips page but there we were and I used it occasionally my self and it seems people were using it from the traffic stats so I guess it was successful

It was at around this time I was approached by Bacon_Space and shortly after AndreAwesome about wanting to help out, I was thrilled because honestly people don't reach out to me much at all and I could definitely use the help.At this point I had added over 1.3k characters and who knows how many streamers all via the Java command line app so someone to help with that data was a godsend. The problem was I didn't have a good way for him to help, he could just message me chars for streamers but that would get old for both of us really quickly. So once again another feature I hadn't planned was required.

Staff features

I had made a blog at some point earlier as my only real form of communication with my users was Reddit posts until then which wasn't sustainable. The admin panel at that point was literally just a blog editor with a hardcoded password hash to verify against. So I said yeah welcome aboard and got to work writing a login system, CSRF system, updated cache systems and write a web user interface for helpers to add characters and edit them data validation all sorts of things we didn't have but now needed.

The web interface really was a godsend as it made my life a lot easier, the site and project are always demanding of my time so most things get put on a list to be addressed one day and most things don't come off those lists. I always prioritised user facing features until this point but a little quality of life upgrade reinvigorated me to starting adding characters again. Since then I have added features to be able to move streamers, force offline streamers for TOS violations for staff, update the character lists as edits are made and many small other things for the sites staff. The system has had many tweaks since creation and is in a pretty great place right now.

The end?

I have sworn many times my next update to the site was going to be the last and at many times major blows have been felt by the GTA RP scene making me reconsider its future. When I started all of this it was during SoE and when the major split happened I felt like I had really wasted a lot of my time but I continued on and followed the scene to the Family RP. I won't go into the details of the drama that happened there but at times it made me question the future of the site and if the scene may just fall apart. When I started tracking more servers even No Pixel has had huge surges and dips. At the time of writing this (April 7th) it is currently receiving a huge surge but like all things that too shall pass. That surge has brought us insane traffic though and also a few DDoS attacks but it has also made me want to work on the site a bit more, I have never really stopped but new features come in surges when I feel motivated.

That being said I don't have any major plans for new features on the horizon, but that is subject to change depending on what I as a viewer think would be cool and what you all as a community suggest. I have had many ideas of ways to take the site in the past: push the community aspect by getting forums and bloggers to write about ongoing RP storylines, our own GTA RP servers, to take it out of just the GTA RP sphere and into all gaming RP, to create a mod for GTA RP servers for more data to be shared that would be useful for this site, track different streaming sites such as mixer or to follow non English RP servers. So far I haven't been motivated enough or haven't felt they could be worked out in a good enough way to be worth the time, money and effort they would take. Maybe one day though.