ATS Technology Update/Info

This morning we encountered some problems that seem to be related to the increased popularity of the MyATS page combined with a couple of other factors.

As it turns out, our unique size and combination of features (such as the ability to run boolean search on posts) is developing into somewhat of a problem that isn't readily solved by server CPU horsepower alone. When you have a 4 gigagyte data set, some things are going to be harder no matter how many CPU-cycles you can throw at the problem.

And just to put things into perspective, irrespective of searches, profile pages, forum pages and MyATS -- we're serving about two thread-pages a second during our slow times, and up to five thread-pages a second during our peak periods... that's a lot of core activity.

For now, some features such as Boolean search and aspects of MyATS will be either disabled or limited. We're seeking out an exceptional database design and optimization expert who can look at our installation, and offer strategies for performance enhancements.

We'll keep everyone updated on the progress.

Ah thanks.. that explains things

50/50

Whatever it takes to keep this place running smoothly!

For members who will flip their lid about the "myATS" page being disabled, please remember that you can customize the recent posts page and use that to keep track of recent activity.
.

Originally posted by SkepticOverlord
We're seeking out an exceptional database design and optimization expert who can look at our installation, and offer strategies for performance enhancements.

I'd offer my services but it would be akin to the work experience kid re wiring the space shuttle!

Keep up the good work ~ carry on

Whatever you gotta do Bill- just don't take away my addiction......

Most problems I noticed aren't really database problems but the way you tell php to get data from the database.

Also, I'm wondering if your using any database caching systems like memcached for queries results. You probably think you don't need if your already using the mysql query cache, but its been discussed many times in the developers community that the query cache is far from optimal.

And even if you use memcached or eaccelerator, its peculiar that some queries (like the subscription page ones) get executed every time you refresh the page.

If it was 1 single big query requesting all the entries, it would be normal for memcached or eaccelerator to not cache the query (since it normally is configured to not store seldom used big result sets.)

But since the page works by requesting each entry with a seperate query (which translates in me getting 1900 to 2000 queries for the subscription page alone) it should cache some of those results (since they are small), which would mean the page should reload much much faster the second time around, which it doesn't.

Also, as I suggested in the thread thats now moved to the mods forum, paginating pages like the subscriptions page (and others where you get a single flat list of results) is one step to get rid of some serious performance hits. You can craft paginated pages so that only the results for that page are requested instead of scanning for all the results and then taking the ones for that page out of them.

There's many more things that can be changed for the better, but I'm afraid alot of them would mean serious rewrites of core parts of the website. And maybe even some refactoring of the database itself.

XMB, which ATS was based of, is one of the forums that uses the least amount of tables to hold its data in, compared to other forum applications.

Apps like VB and SMF use extra tables to hold much requested statistical like data in so that operations like requesting all threads a user posted in gets called from that extra table, instead of needing to scan the entire threads and messages tables to compile those results(you might have already changed some things in that respect)

Its less of a performance impact to write something extra to another table or 2, when a new post is made, then having to call all data from 1 single huge messages table afterwards.

There's alot more that can be done and some things that absolutely need to be done.

And after they are done you might come to the realization that some of the design problems can't be solved by trowing hardware at them. And also that once you streamlined the sites engine, you find that the hardware you have is a bit overkill compared to what you actually need.

SO

Just to notify you twice this has logged me on as other people , allowing me to see other people sATs pages and there info , i hope it wont allow you to [post while logged on as these others that could lead to false posts disasters.. just a heads up.

it has logged me on as illusionix and tarzanbeta so far....i could for instance go to there myats store and blow all there points...although of course i would not

reply to post by Quantum_Squirrel


What ISP are you with?

This kind of thing is typical for people on AOL and other ISP's that use overly zealous proxy systems that cache everything including cookie data.

Originally posted by thematrix
XMB, which ATS was based of, is one of the forums that uses the least amount of tables to hold its data in, compared to other forum applications.

There's very little legacy of the old XMB left, both in code and database.

While relational tables can provide some performance improvements, the bottom line is that 98.5% of all our queries are related to pulling posts to build thread pages. In that regard, well-structured indexes and well-designed queries that take advantage of a good index is the most important thing.

I'm sure there is a long list of incremental tweaks to the server OS, MySQL set-up, and table structure... all of which would add up to performance improvements. But some of the things we built when we had less than 1 million posts (like a full-text index on all posts to accommodate boolean search) are built well, but present problems no matter how much horsepower and server optimization is applied.

The last time a db expert reviewed our basic installation (right after the recent transition), he was impressed with the overall table structure and indexes and felt the core code/structure was well-suited to our expected traffic. However, he was more of a server expert and not so much an application expert.

Good to hear that the board has evolved far away from its XMB roots SO.
XMB was recently taken over by a new group of developers and the last several releases have all been bug fixes to resolve serious exploits, even though XMB is currently versioned 1.9.7 (and alot of the bugs and exploits in it were what you typicaly find in pre-V1 software).

Without knowing how and to what extend the database and code have evolved from their original roots, its hard to suggest anything other then the obvious problems I witnessed.

Stuff like the subscriptions page, even though its probably not used often, is the kind of thing what can bring an otherwise streamlined and well designed site, down on its knees, no matter what hardware you trow at it, just because it mixes in with the normal operations from time to time and then causes things to go awry.

As they say, a chain is only as strong as its weakest link.

I myself would be considered a "server expert" too, but for me personally that means knowing and understanding the interaction between server and its applications too.
I might not be able to develop software or web-apps from the ground up like software developers can, but I can read and write code perfectly, I'm rather anal about OO development and adherence to standards and usually spot bugs and design flaws that the programmers themselves don't.

Also, I wonder if you ever thought about making a true redesign happen for ATS's website, instead of just evolving the site to new version levels.

The people that created Django for instance, they worked for a newspaper website where everything was running on PHP and because they found themselves in an ever expanding flood of fixing bugs that creaped out of old parts of code, they did a load of research and eventually designed the Django Web Framework (in python) to suit their needs and wants.

They crafted certain parts to interact with the old data in an entirely new way while developing completely new feature sets for everything else.

Doing something like this, off course would be a pretty massive undertaking, but doing it as sort of a side project until it reaches sufficient maturity to replace the old platform could be best in the long run.

You can't keep patching and updating old foundations and design philosophies. At one point you have to set a vision of how you like to see things and reinvent your application from scratch to meet that vision.

What a shame! I really LIKE the "my ATS" feature. It's tough remembering all the threads you had an interest in, and I feel it gives the threads more life.

Otherwise I'm afraid they'll degenerate into "drive-by postings" - all comment and no discussion.

MyATS is returned with a slightly refined query process.

Another item that was noticed today... at some point after the transition, the Google crawler picked up the performance boost, and began crawling the site more often... we were up to an average of 550,000 pages a day. While these don't count in our traffic logging, that's a lot of activity to support. I recently used the Google webmaster tools to throttle it back to "normal" crawl speed.

Keep in mind that system bus bandwidth is more crucial than extra CPU horsepower as you say.

I for one like the new MyATS page, the within 30 day limit is good. I hate clearing up old subscriptions.

Thanks

/edit to correct a number

[edit on 28/8/07 by SteveR]

Originally posted by SteveR
Keep in mind that system bus bandwidth is more crucial than extra CPU horsepower as you say.

We got a wide bus.
And also very-fast hard drives and RAM.
And our server cluster is on an optical gigabit private network.

Originally posted by Gools
Whatever it takes to keep this place running smoothly!

For members who will flip their lid about the "myATS" page being disabled.

It hasn't been disabled, merely cut to the most recent 30 days.

EDIT: See Implosion's response.

I understand there is a lot of lag on this place due to all the new features and amount of bandwidth being pumped through the servers.

[edit on 28-8-2007 by biggie smalls]

Biggie, I'm not sure, but I think you might still be able to find those threads using this link. Could be wrong though, apologies if I am.

[edit on 28/8/07 by Implosion]

Whatever it takes, SO... We can be patient for a while. I'm sure it won't kill us. Keep up the good work.

Tell the truth..your installing secret backdoors for the CIA!!! (joke)

Long as ATS keeps running I for one don't mind occasional thingies like this while improvements are made. Its good you've kept users informed though.


Wayne...

i like the way things are going so far keep up the good work.

hehe